Wed Mar 20 19:53:54.832 [initandlisten] MongoDB starting : pid=20398 port=27017 dbpath=/var/lib/mongodb 64-bit host=db20 Wed Mar 20 19:53:54.832 [initandlisten] db version v2.4.0 Wed Mar 20 19:53:54.832 [initandlisten] git version: ce2d666c04b4a80af58e8bbb3388b0680e8cfeb6 Wed Mar 20 19:53:54.832 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49 Wed Mar 20 19:53:54.832 [initandlisten] allocator: tcmalloc Wed Mar 20 19:53:54.832 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log" } Wed Mar 20 19:53:54.844 [initandlisten] journal dir=/var/lib/mongodb/journal Wed Mar 20 19:53:54.844 [initandlisten] recover : no journal files present, no recovery needed Wed Mar 20 19:53:54.981 [FileAllocator] allocating new datafile /var/lib/mongodb/local.ns, filling with zeroes... Wed Mar 20 19:53:54.981 [FileAllocator] creating directory /var/lib/mongodb/_tmp Wed Mar 20 19:53:54.984 [FileAllocator] done allocating datafile /var/lib/mongodb/local.ns, size: 16MB, took 0.001 secs Wed Mar 20 19:53:54.985 [FileAllocator] allocating new datafile /var/lib/mongodb/local.0, filling with zeroes... Wed Mar 20 19:53:55.161 [FileAllocator] done allocating datafile /var/lib/mongodb/local.0, size: 64MB, took 0.176 secs Wed Mar 20 19:53:55.170 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 189ms Wed Mar 20 19:53:55.171 [initandlisten] waiting for connections on port 27017 Wed Mar 20 19:53:55.171 [websvr] admin web console waiting for connections on port 28017 Wed Mar 20 19:53:58.141 got signal 2 (Interrupt), will terminate after current cmd ends Wed Mar 20 19:53:58.141 [interruptThread] now exiting Wed Mar 20 19:53:58.141 dbexit: Wed Mar 20 19:53:58.141 [interruptThread] shutdown: going to close listening sockets... Wed Mar 20 19:53:58.141 [interruptThread] closing listening socket: 9 Wed Mar 20 19:53:58.141 [interruptThread] closing listening socket: 10 Wed Mar 20 19:53:58.141 [interruptThread] closing listening socket: 11 Wed Mar 20 19:53:58.141 [interruptThread] removing socket file: /tmp/mongodb-27017.sock Wed Mar 20 19:53:58.141 [interruptThread] shutdown: going to flush diaglog... Wed Mar 20 19:53:58.141 [interruptThread] shutdown: going to close sockets... Wed Mar 20 19:53:58.142 [interruptThread] shutdown: waiting for fs preallocator... Wed Mar 20 19:53:58.142 [interruptThread] shutdown: lock for final commit... Wed Mar 20 19:53:58.142 [interruptThread] shutdown: final commit... Wed Mar 20 19:53:58.150 [interruptThread] shutdown: closing all files... Wed Mar 20 19:53:58.150 [interruptThread] closeAllFiles() finished Wed Mar 20 19:53:58.150 [interruptThread] journalCleanup... Wed Mar 20 19:53:58.150 [interruptThread] removeJournalFiles Wed Mar 20 19:53:58.156 [interruptThread] shutdown: removing fs lock... Wed Mar 20 19:53:58.156 dbexit: really exiting now ***** SERVER RESTARTED ***** Wed Mar 20 19:58:22.052 [initandlisten] MongoDB starting : pid=1940 port=27017 dbpath=/var/lib/mongodb 64-bit host=db20 Wed Mar 20 19:58:22.053 [initandlisten] db version v2.4.0 Wed Mar 20 19:58:22.053 [initandlisten] git version: ce2d666c04b4a80af58e8bbb3388b0680e8cfeb6 Wed Mar 20 19:58:22.053 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49 Wed Mar 20 19:58:22.053 [initandlisten] allocator: tcmalloc Wed Mar 20 19:58:22.053 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log" } Wed Mar 20 19:58:22.063 [initandlisten] journal dir=/var/lib/mongodb/journal Wed Mar 20 19:58:22.063 [initandlisten] recover : no journal files present, no recovery needed Wed Mar 20 19:58:22.219 [websvr] admin web console waiting for connections on port 28017 Wed Mar 20 19:58:22.219 [initandlisten] waiting for connections on port 27017 Wed Mar 20 20:05:59.568 got signal 15 (Terminated), will terminate after current cmd ends Wed Mar 20 20:05:59.568 [interruptThread] now exiting Wed Mar 20 20:05:59.568 dbexit: Wed Mar 20 20:05:59.568 [interruptThread] shutdown: going to close listening sockets... Wed Mar 20 20:05:59.568 [interruptThread] closing listening socket: 9 Wed Mar 20 20:05:59.568 [interruptThread] closing listening socket: 10 Wed Mar 20 20:05:59.568 [interruptThread] closing listening socket: 11 Wed Mar 20 20:05:59.568 [interruptThread] removing socket file: /tmp/mongodb-27017.sock Wed Mar 20 20:05:59.568 [interruptThread] shutdown: going to flush diaglog... Wed Mar 20 20:05:59.568 [interruptThread] shutdown: going to close sockets... Wed Mar 20 20:05:59.569 [interruptThread] shutdown: waiting for fs preallocator... Wed Mar 20 20:05:59.569 [interruptThread] shutdown: lock for final commit... Wed Mar 20 20:05:59.569 [interruptThread] shutdown: final commit... Wed Mar 20 20:05:59.569 [interruptThread] shutdown: closing all files... Wed Mar 20 20:05:59.570 [interruptThread] closeAllFiles() finished Wed Mar 20 20:05:59.570 [interruptThread] journalCleanup... Wed Mar 20 20:05:59.570 [interruptThread] removeJournalFiles Wed Mar 20 20:05:59.575 [interruptThread] shutdown: removing fs lock... Wed Mar 20 20:05:59.576 dbexit: really exiting now ***** SERVER RESTARTED ***** Wed Mar 20 20:06:09.660 [initandlisten] MongoDB starting : pid=6103 port=27017 dbpath=/var/lib/mongodb 64-bit host=db20 Wed Mar 20 20:06:09.660 [initandlisten] db version v2.4.0 Wed Mar 20 20:06:09.660 [initandlisten] git version: ce2d666c04b4a80af58e8bbb3388b0680e8cfeb6 Wed Mar 20 20:06:09.660 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49 Wed Mar 20 20:06:09.660 [initandlisten] allocator: tcmalloc Wed Mar 20 20:06:09.660 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log", noauth: "true", replSet: "nbset", rest: "true", slowms: 100 } Wed Mar 20 20:06:09.670 [initandlisten] journal dir=/var/lib/mongodb/journal Wed Mar 20 20:06:09.671 [initandlisten] recover : no journal files present, no recovery needed Wed Mar 20 20:06:09.812 [initandlisten] waiting for connections on port 27017 Wed Mar 20 20:06:09.813 [websvr] admin web console waiting for connections on port 28017 Wed Mar 20 20:06:09.819 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Wed Mar 20 20:06:09.819 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done Wed Mar 20 20:06:19.820 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Wed Mar 20 20:06:29.820 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Wed Mar 20 20:06:39.821 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Wed Mar 20 20:06:49.821 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Wed Mar 20 20:06:59.822 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Wed Mar 20 20:07:09.847 [rsStart] trying to contact db05:27017 Wed Mar 20 20:07:09.913 [rsStart] replSet I am db20:27017 Wed Mar 20 20:07:09.913 [rsStart] replSet got config version 335595 from a remote, saving locally Wed Mar 20 20:07:09.913 [rsStart] replSet info saving a newer config version to local.system.replset Wed Mar 20 20:07:09.916 [rsStart] replSet saveConfigLocally done Wed Mar 20 20:07:09.917 [rsStart] replSet STARTUP2 Wed Mar 20 20:07:09.918 [rsSync] ****** Wed Mar 20 20:07:09.918 [rsSync] creating replication oplog of size: 15207MB... Wed Mar 20 20:07:09.918 [FileAllocator] allocating new datafile /var/lib/mongodb/local.1, filling with zeroes... Wed Mar 20 20:07:09.918 [FileAllocator] creating directory /var/lib/mongodb/_tmp Wed Mar 20 20:07:09.930 [FileAllocator] done allocating datafile /var/lib/mongodb/local.1, size: 2047MB, took 0.007 secs Wed Mar 20 20:07:09.931 [FileAllocator] allocating new datafile /var/lib/mongodb/local.2, filling with zeroes... Wed Mar 20 20:07:09.937 [FileAllocator] done allocating datafile /var/lib/mongodb/local.2, size: 2047MB, took 0.006 secs Wed Mar 20 20:07:09.938 [FileAllocator] allocating new datafile /var/lib/mongodb/local.3, filling with zeroes... Wed Mar 20 20:07:09.945 [FileAllocator] done allocating datafile /var/lib/mongodb/local.3, size: 2047MB, took 0.006 secs Wed Mar 20 20:07:09.946 [FileAllocator] allocating new datafile /var/lib/mongodb/local.4, filling with zeroes... Wed Mar 20 20:07:09.952 [FileAllocator] done allocating datafile /var/lib/mongodb/local.4, size: 2047MB, took 0.006 secs Wed Mar 20 20:07:09.953 [FileAllocator] allocating new datafile /var/lib/mongodb/local.5, filling with zeroes... Wed Mar 20 20:07:09.959 [FileAllocator] done allocating datafile /var/lib/mongodb/local.5, size: 2047MB, took 0.006 secs Wed Mar 20 20:07:09.960 [FileAllocator] allocating new datafile /var/lib/mongodb/local.6, filling with zeroes... Wed Mar 20 20:07:09.967 [FileAllocator] done allocating datafile /var/lib/mongodb/local.6, size: 2047MB, took 0.006 secs Wed Mar 20 20:07:09.968 [FileAllocator] allocating new datafile /var/lib/mongodb/local.7, filling with zeroes... Wed Mar 20 20:07:09.976 [FileAllocator] done allocating datafile /var/lib/mongodb/local.7, size: 2047MB, took 0.007 secs Wed Mar 20 20:07:09.977 [FileAllocator] allocating new datafile /var/lib/mongodb/local.8, filling with zeroes... Wed Mar 20 20:07:09.983 [FileAllocator] done allocating datafile /var/lib/mongodb/local.8, size: 2047MB, took 0.006 secs Wed Mar 20 20:07:09.984 [rsSync] ****** Wed Mar 20 20:07:09.985 [rsSync] replSet initial sync pending Wed Mar 20 20:07:09.985 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync Wed Mar 20 20:07:11.925 [rsHealthPoll] replSet member db05:27017 is up Wed Mar 20 20:07:11.925 [rsHealthPoll] replSet member db05:27017 is now in state SECONDARY Wed Mar 20 20:07:11.936 [rsHealthPoll] replSet member db04:27017 is up Wed Mar 20 20:07:11.936 [rsHealthPoll] replSet member db04:27017 is now in state PRIMARY Wed Mar 20 20:07:25.985 [rsSync] replSet initial sync pending Wed Mar 20 20:07:25.985 [rsSync] replSet syncing to: db04:27017 Wed Mar 20 20:07:26.000 [rsSync] build index local.me { _id: 1 } Wed Mar 20 20:07:26.002 [rsSync] build index done. scanned 0 total records. 0.001 secs Wed Mar 20 20:07:26.055 [rsSync] build index local.replset.minvalid { _id: 1 } Wed Mar 20 20:07:26.056 [rsSync] build index done. scanned 0 total records. 0 secs Wed Mar 20 20:07:26.056 [rsSync] replSet initial sync drop all databases Wed Mar 20 20:07:26.056 [rsSync] dropAllDatabasesExceptLocal 1 Wed Mar 20 20:07:26.056 [rsSync] replSet initial sync clone all databases Wed Mar 20 20:07:26.087 [rsSync] replSet initial sync cloning db: admin Wed Mar 20 20:07:26.133 [FileAllocator] allocating new datafile /var/lib/mongodb/admin.ns, filling with zeroes... Wed Mar 20 20:07:26.136 [FileAllocator] done allocating datafile /var/lib/mongodb/admin.ns, size: 16MB, took 0.002 secs Wed Mar 20 20:07:26.137 [FileAllocator] allocating new datafile /var/lib/mongodb/admin.0, filling with zeroes... Wed Mar 20 20:07:26.304 [FileAllocator] done allocating datafile /var/lib/mongodb/admin.0, size: 64MB, took 0.167 secs Wed Mar 20 20:07:26.306 [FileAllocator] allocating new datafile /var/lib/mongodb/admin.1, filling with zeroes... Wed Mar 20 20:07:26.308 [FileAllocator] done allocating datafile /var/lib/mongodb/admin.1, size: 128MB, took 0.002 secs Wed Mar 20 20:07:26.314 [rsSync] build index admin.system.users { user: 1, userSource: 1 } Wed Mar 20 20:07:26.317 [rsSync] build index done. scanned 0 total records. 0.002 secs Wed Mar 20 20:07:26.433 [rsSync] build index admin.system.users { _id: 1 } Wed Mar 20 20:07:26.437 [rsSync] fastBuildIndex dupsToDrop:0 Wed Mar 20 20:07:26.437 [rsSync] build index done. scanned 1 total records. 0.003 secs Wed Mar 20 20:07:26.437 [rsSync] replSet initial sync cloning db: newsblur Wed Mar 20 20:07:26.896 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.ns, filling with zeroes... Wed Mar 20 20:07:26.901 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.ns, size: 16MB, took 0.004 secs Wed Mar 20 20:07:26.902 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.0, filling with zeroes... Wed Mar 20 20:07:26.905 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.0, size: 64MB, took 0.002 secs Wed Mar 20 20:07:26.906 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.1, filling with zeroes... Wed Mar 20 20:07:26.908 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.1, size: 128MB, took 0.002 secs Wed Mar 20 20:07:56.048 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.2, filling with zeroes... Wed Mar 20 20:07:56.052 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.2, size: 256MB, took 0.003 secs Wed Mar 20 20:08:27.525 [rsSync] 130048 objects cloned so far from collection newsblur.stories Wed Mar 20 20:08:30.564 [rsSync] clone newsblur.stories 135423 Wed Mar 20 20:08:49.942 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.3, filling with zeroes... Wed Mar 20 20:08:49.947 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.3, size: 512MB, took 0.004 secs Wed Mar 20 20:09:28.425 [rsSync] 288290 objects cloned so far from collection newsblur.stories Wed Mar 20 20:09:30.859 [rsSync] clone newsblur.stories 293631 Wed Mar 20 20:09:58.634 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.4, filling with zeroes... Wed Mar 20 20:09:58.638 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.4, size: 1024MB, took 0.003 secs Wed Mar 20 20:10:29.310 [rsSync] 438658 objects cloned so far from collection newsblur.stories Wed Mar 20 20:10:30.451 [rsSync] clone newsblur.stories 443391 Wed Mar 20 20:11:30.396 [rsSync] 606806 objects cloned so far from collection newsblur.stories Wed Mar 20 20:11:30.398 [rsSync] clone newsblur.stories 606847 Wed Mar 20 20:12:30.339 [rsSync] clone newsblur.stories 753791 Wed Mar 20 20:12:31.082 [rsSync] 757555 objects cloned so far from collection newsblur.stories Wed Mar 20 20:13:14.742 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.5, filling with zeroes... Wed Mar 20 20:13:14.746 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.5, size: 2047MB, took 0.003 secs Wed Mar 20 20:13:30.521 [rsSync] clone newsblur.stories 873855 Wed Mar 20 20:13:32.322 [rsSync] 877570 objects cloned so far from collection newsblur.stories Wed Mar 20 20:14:30.170 [rsSync] clone newsblur.stories 996479 Wed Mar 20 20:14:33.713 [rsSync] 1000125 objects cloned so far from collection newsblur.stories Wed Mar 20 20:15:30.949 [rsSync] clone newsblur.stories 1093631 Wed Mar 20 20:15:34.605 [rsSync] 1101471 objects cloned so far from collection newsblur.stories Wed Mar 20 20:16:30.441 [rsSync] clone newsblur.stories 1202303 Wed Mar 20 20:16:36.515 [rsSync] 1216152 objects cloned so far from collection newsblur.stories Wed Mar 20 20:17:10.603 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.6, filling with zeroes... Wed Mar 20 20:17:10.609 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.6, size: 2047MB, took 0.005 secs Wed Mar 20 20:17:31.021 [rsSync] clone newsblur.stories 1337215 Wed Mar 20 20:17:37.333 [rsSync] 1349629 objects cloned so far from collection newsblur.stories Wed Mar 20 20:18:31.721 [rsSync] clone newsblur.stories 1481983 Wed Mar 20 20:18:40.143 [rsSync] 1506474 objects cloned so far from collection newsblur.stories Wed Mar 20 20:19:31.504 [rsSync] clone newsblur.stories 1638783 Wed Mar 20 20:19:41.183 [rsSync] 1664462 objects cloned so far from collection newsblur.stories Wed Mar 20 20:20:32.195 [rsSync] clone newsblur.stories 1757311 Wed Mar 20 20:20:42.650 [rsSync] 1789303 objects cloned so far from collection newsblur.stories Wed Mar 20 20:21:32.973 [rsSync] clone newsblur.stories 1902719 Wed Mar 20 20:21:43.179 [rsSync] 1928241 objects cloned so far from collection newsblur.stories Wed Mar 20 20:22:33.117 [rsSync] clone newsblur.stories 2039807 Wed Mar 20 20:22:44.122 [rsSync] 2078228 objects cloned so far from collection newsblur.stories Wed Mar 20 20:23:33.141 [rsSync] clone newsblur.stories 2186111 Wed Mar 20 20:23:46.086 [rsSync] 2219960 objects cloned so far from collection newsblur.stories Wed Mar 20 20:24:07.704 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.7, filling with zeroes... Wed Mar 20 20:24:07.709 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.7, size: 2047MB, took 0.004 secs Wed Mar 20 20:24:35.107 [rsSync] clone newsblur.stories 2316927 Wed Mar 20 20:24:49.480 [rsSync] 2325116 objects cloned so far from collection newsblur.stories Wed Mar 20 20:25:35.249 [rsSync] clone newsblur.stories 2391935 Wed Mar 20 20:25:44.763 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._0 Wed Mar 20 20:25:50.763 [rsSync] 2429637 objects cloned so far from collection newsblur.stories Wed Mar 20 20:26:36.488 [rsSync] clone newsblur.stories 2523007 Wed Mar 20 20:26:52.532 [rsSync] 2559939 objects cloned so far from collection newsblur.stories Wed Mar 20 20:27:38.435 [rsSync] clone newsblur.stories 2637823 Wed Mar 20 20:27:53.481 [rsSync] 2677541 objects cloned so far from collection newsblur.stories Wed Mar 20 20:28:38.193 [rsSync] clone newsblur.stories 2785407 Wed Mar 20 20:28:54.727 [rsSync] 2829814 objects cloned so far from collection newsblur.stories Wed Mar 20 20:29:38.260 [rsSync] clone newsblur.stories 2939647 Wed Mar 20 20:29:55.102 [rsSync] 2981706 objects cloned so far from collection newsblur.stories Wed Mar 20 20:30:38.958 [rsSync] clone newsblur.stories 3054975 Wed Mar 20 20:30:39.037 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.8, filling with zeroes... Wed Mar 20 20:30:39.043 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.8, size: 2047MB, took 0.005 secs Wed Mar 20 20:30:56.005 [rsSync] 3091561 objects cloned so far from collection newsblur.stories Wed Mar 20 20:31:39.879 [rsSync] clone newsblur.stories 3106303 Wed Mar 20 20:32:05.417 [rsSync] 3122549 objects cloned so far from collection newsblur.stories Wed Mar 20 20:32:39.511 [rsSync] clone newsblur.stories 3184255 Wed Mar 20 20:33:06.464 [rsSync] 3251073 objects cloned so far from collection newsblur.stories Wed Mar 20 20:33:39.987 [rsSync] clone newsblur.stories 3309951 Wed Mar 20 20:34:07.623 [rsSync] 3360547 objects cloned so far from collection newsblur.stories Wed Mar 20 20:34:39.271 [rsSync] clone newsblur.stories 3408511 Wed Mar 20 20:35:08.206 [rsSync] 3463417 objects cloned so far from collection newsblur.stories Wed Mar 20 20:35:32.479 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._1 Wed Mar 20 20:35:39.527 [rsSync] clone newsblur.stories 3521023 Wed Mar 20 20:36:10.503 [rsSync] 3573399 objects cloned so far from collection newsblur.stories Wed Mar 20 20:36:39.764 [rsSync] clone newsblur.stories 3635455 Wed Mar 20 20:37:11.102 [rsSync] 3700653 objects cloned so far from collection newsblur.stories Wed Mar 20 20:37:40.460 [rsSync] clone newsblur.stories 3758207 Wed Mar 20 20:38:13.279 [rsSync] 3825240 objects cloned so far from collection newsblur.stories Wed Mar 20 20:38:42.515 [rsSync] clone newsblur.stories 3891199 Wed Mar 20 20:39:13.396 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.9, filling with zeroes... Wed Mar 20 20:39:13.402 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.9, size: 2047MB, took 0.005 secs Wed Mar 20 20:39:14.929 [rsSync] 3976624 objects cloned so far from collection newsblur.stories Wed Mar 20 20:39:42.634 [rsSync] clone newsblur.stories 4020607 Wed Mar 20 20:40:16.843 [rsSync] 4035298 objects cloned so far from collection newsblur.stories Wed Mar 20 20:40:43.342 [rsSync] clone newsblur.stories 4055807 Wed Mar 20 20:41:21.154 [rsSync] 4096766 objects cloned so far from collection newsblur.stories Wed Mar 20 20:41:45.833 [rsSync] clone newsblur.stories 4136831 Wed Mar 20 20:42:23.568 [rsSync] 4217543 objects cloned so far from collection newsblur.stories Wed Mar 20 20:42:45.855 [rsSync] clone newsblur.stories 4273535 Wed Mar 20 20:43:27.943 [rsSync] 4354830 objects cloned so far from collection newsblur.stories Wed Mar 20 20:43:46.013 [rsSync] clone newsblur.stories 4389247 Wed Mar 20 20:44:28.390 [rsSync] 4469130 objects cloned so far from collection newsblur.stories Wed Mar 20 20:44:33.101 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._2 Wed Mar 20 20:44:46.626 [rsSync] clone newsblur.stories 4511999 Wed Mar 20 20:45:29.315 [rsSync] 4586605 objects cloned so far from collection newsblur.stories Wed Mar 20 20:45:46.083 [rsSync] clone newsblur.stories 4637823 Wed Mar 20 20:46:30.041 [rsSync] 4736338 objects cloned so far from collection newsblur.stories Wed Mar 20 20:46:46.376 [rsSync] clone newsblur.stories 4785663 Wed Mar 20 20:47:32.127 [rsSync] 4924171 objects cloned so far from collection newsblur.stories Wed Mar 20 20:47:47.084 [rsSync] clone newsblur.stories 4971135 Wed Mar 20 20:48:34.704 [rsSync] 5079365 objects cloned so far from collection newsblur.stories Wed Mar 20 20:48:49.305 [rsSync] clone newsblur.stories 5107071 Wed Mar 20 20:49:35.415 [rsSync] 5191252 objects cloned so far from collection newsblur.stories Wed Mar 20 20:49:49.469 [rsSync] clone newsblur.stories 5236991 Wed Mar 20 20:49:53.345 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.10, filling with zeroes... Wed Mar 20 20:49:53.349 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.10, size: 2047MB, took 0.003 secs Wed Mar 20 20:50:36.432 [rsSync] 5337810 objects cloned so far from collection newsblur.stories Wed Mar 20 20:50:49.262 [rsSync] clone newsblur.stories 5345151 Wed Mar 20 20:51:42.822 [rsSync] 5368417 objects cloned so far from collection newsblur.stories Wed Mar 20 20:51:50.872 [rsSync] clone newsblur.stories 5376895 Wed Mar 20 20:52:45.321 [rsSync] 5430774 objects cloned so far from collection newsblur.stories Wed Mar 20 20:52:53.519 [rsSync] clone newsblur.stories 5445887 Wed Mar 20 20:53:15.891 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._3 Wed Mar 20 20:53:15.914 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._4 Wed Mar 20 20:53:46.467 [rsSync] 5558282 objects cloned so far from collection newsblur.stories Wed Mar 20 20:53:54.046 [rsSync] clone newsblur.stories 5576959 Wed Mar 20 20:54:48.164 [rsSync] 5697093 objects cloned so far from collection newsblur.stories Wed Mar 20 20:54:54.264 [rsSync] clone newsblur.stories 5709823 Wed Mar 20 20:55:51.152 [rsSync] 5838190 objects cloned so far from collection newsblur.stories Wed Mar 20 20:55:54.006 [rsSync] clone newsblur.stories 5847551 Wed Mar 20 20:56:53.702 [rsSync] 5974969 objects cloned so far from collection newsblur.stories Wed Mar 20 20:56:54.625 [rsSync] clone newsblur.stories 5979135 Wed Mar 20 20:57:56.317 [rsSync] 6147054 objects cloned so far from collection newsblur.stories Wed Mar 20 20:57:56.318 [rsSync] clone newsblur.stories 6147071 Wed Mar 20 20:58:56.243 [initandlisten] connection accepted from 198.211.115.153:46525 #7908 (19 connections now open) Wed Mar 20 20:58:56.672 [rsSync] clone newsblur.stories 6291583 Wed Mar 20 20:58:57.410 [rsSync] 6294374 objects cloned so far from collection newsblur.stories Wed Mar 20 20:59:26.305 [initandlisten] connection accepted from 198.211.115.153:46537 #7989 (20 connections now open) Wed Mar 20 20:59:56.791 [rsSync] clone newsblur.stories 6449791 Wed Mar 20 21:00:02.223 [rsSync] 6455664 objects cloned so far from collection newsblur.stories Wed Mar 20 21:00:57.516 [rsSync] clone newsblur.stories 6570239 Wed Mar 20 21:01:03.261 [rsSync] 6584950 objects cloned so far from collection newsblur.stories Wed Mar 20 21:01:57.040 [rsSync] clone newsblur.stories 6675583 Wed Mar 20 21:02:04.493 [rsSync] 6691757 objects cloned so far from collection newsblur.stories Wed Mar 20 21:02:57.780 [rsSync] clone newsblur.stories 6778111 Wed Mar 20 21:03:05.375 [rsSync] 6792468 objects cloned so far from collection newsblur.stories Wed Mar 20 21:03:31.567 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.11, filling with zeroes... Wed Mar 20 21:03:31.573 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.11, size: 2047MB, took 0.005 secs Wed Mar 20 21:03:57.791 [rsSync] clone newsblur.stories 6923391 Wed Mar 20 21:04:11.156 [rsSync] 6939153 objects cloned so far from collection newsblur.stories Wed Mar 20 21:04:58.657 [rsSync] clone newsblur.stories 6959231 Wed Mar 20 21:05:14.258 [rsSync] 6971157 objects cloned so far from collection newsblur.stories Wed Mar 20 21:05:59.185 [rsSync] clone newsblur.stories 7005183 Wed Mar 20 21:06:16.854 [rsSync] 7022211 objects cloned so far from collection newsblur.stories Wed Mar 20 21:06:59.390 [rsSync] clone newsblur.stories 7128063 Wed Mar 20 21:07:18.344 [rsSync] 7195566 objects cloned so far from collection newsblur.stories Wed Mar 20 21:07:59.886 [rsSync] clone newsblur.stories 7291775 Wed Mar 20 21:08:20.638 [rsSync] 7345680 objects cloned so far from collection newsblur.stories Wed Mar 20 21:08:59.236 [rsSync] clone newsblur.stories 7422463 Wed Mar 20 21:09:21.242 [rsSync] 7462393 objects cloned so far from collection newsblur.stories Wed Mar 20 21:10:00.022 [rsSync] clone newsblur.stories 7545983 Wed Mar 20 21:10:01.083 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._5 Wed Mar 20 21:10:25.779 [rsSync] 7593457 objects cloned so far from collection newsblur.stories Wed Mar 20 21:11:00.015 [rsSync] clone newsblur.stories 7686143 Wed Mar 20 21:11:28.201 [rsSync] 7748397 objects cloned so far from collection newsblur.stories Wed Mar 20 21:12:01.350 [rsSync] clone newsblur.stories 7817727 Wed Mar 20 21:12:29.749 [rsSync] 7897241 objects cloned so far from collection newsblur.stories Wed Mar 20 21:13:03.824 [rsSync] clone newsblur.stories 7966335 Wed Mar 20 21:13:30.328 [rsSync] 8014462 objects cloned so far from collection newsblur.stories Wed Mar 20 21:14:05.957 [rsSync] clone newsblur.stories 8104063 Wed Mar 20 21:14:27.295 [conn7908] end connection 198.211.115.153:46525 (19 connections now open) Wed Mar 20 21:14:27.310 [conn7989] end connection 198.211.115.153:46537 (18 connections now open) Wed Mar 20 21:14:31.262 [rsSync] 8147939 objects cloned so far from collection newsblur.stories Wed Mar 20 21:15:05.007 [rsSync] clone newsblur.stories 8237823 Wed Mar 20 21:15:33.689 [rsSync] 8284718 objects cloned so far from collection newsblur.stories Wed Mar 20 21:16:05.257 [rsSync] clone newsblur.stories 8369663 Wed Mar 20 21:16:20.106 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.12, filling with zeroes... Wed Mar 20 21:16:20.113 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.12, size: 2047MB, took 0.006 secs Wed Mar 20 21:16:34.058 [rsSync] 8429879 objects cloned so far from collection newsblur.stories Wed Mar 20 21:17:06.402 [rsSync] clone newsblur.stories 8502783 Wed Mar 20 21:17:39.942 [rsSync] 8515858 objects cloned so far from collection newsblur.stories Wed Mar 20 21:18:07.067 [rsSync] clone newsblur.stories 8536831 Wed Mar 20 21:18:41.635 [rsSync] 8567039 objects cloned so far from collection newsblur.stories Wed Mar 20 21:18:50.511 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._6 Wed Mar 20 21:19:08.818 [rsSync] clone newsblur.stories 8602367 Wed Mar 20 21:19:42.824 [rsSync] 8676005 objects cloned so far from collection newsblur.stories Wed Mar 20 21:20:08.103 [rsSync] clone newsblur.stories 8722303 Wed Mar 20 21:20:44.593 [rsSync] 8800465 objects cloned so far from collection newsblur.stories Wed Mar 20 21:21:09.145 [rsSync] clone newsblur.stories 8871551 Wed Mar 20 21:21:45.569 [rsSync] 8967978 objects cloned so far from collection newsblur.stories Wed Mar 20 21:22:09.609 [rsSync] clone newsblur.stories 9009791 Wed Mar 20 21:22:50.593 [rsSync] 9069705 objects cloned so far from collection newsblur.stories Wed Mar 20 21:23:09.094 [rsSync] clone newsblur.stories 9129599 Wed Mar 20 21:23:53.141 [rsSync] 9208008 objects cloned so far from collection newsblur.stories Wed Mar 20 21:24:09.903 [rsSync] clone newsblur.stories 9249663 Wed Mar 20 21:24:54.581 [rsSync] 9377553 objects cloned so far from collection newsblur.stories Wed Mar 20 21:25:09.424 [rsSync] clone newsblur.stories 9408383 Wed Mar 20 21:25:55.550 [rsSync] 9491472 objects cloned so far from collection newsblur.stories Wed Mar 20 21:26:06.837 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._7 Wed Mar 20 21:26:09.194 [rsSync] clone newsblur.stories 9510143 Wed Mar 20 21:26:56.400 [rsSync] 9588246 objects cloned so far from collection newsblur.stories Wed Mar 20 21:27:09.658 [rsSync] clone newsblur.stories 9607807 Wed Mar 20 21:27:57.665 [rsSync] 9642221 objects cloned so far from collection newsblur.stories Wed Mar 20 21:28:09.360 [rsSync] clone newsblur.stories 9661567 Wed Mar 20 21:28:59.000 [rsSync] 9766046 objects cloned so far from collection newsblur.stories Wed Mar 20 21:29:09.632 [rsSync] clone newsblur.stories 9787263 Wed Mar 20 21:30:00.201 [rsSync] 9855581 objects cloned so far from collection newsblur.stories Wed Mar 20 21:30:02.218 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.13, filling with zeroes... Wed Mar 20 21:30:02.226 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.13, size: 2047MB, took 0.007 secs Wed Mar 20 21:30:09.577 [rsSync] clone newsblur.stories 9864191 Wed Mar 20 21:31:03.925 [rsSync] 9892284 objects cloned so far from collection newsblur.stories Wed Mar 20 21:31:09.890 [rsSync] clone newsblur.stories 9897855 Wed Mar 20 21:32:04.417 [rsSync] 9974961 objects cloned so far from collection newsblur.stories Wed Mar 20 21:32:09.007 [rsSync] clone newsblur.stories 9990527 Wed Mar 20 21:33:06.573 [rsSync] 10160000 objects cloned so far from collection newsblur.stories Wed Mar 20 21:33:09.069 [rsSync] clone newsblur.stories 10170367 Wed Mar 20 21:34:07.410 [rsSync] 10407777 objects cloned so far from collection newsblur.stories Wed Mar 20 21:34:09.530 [rsSync] clone newsblur.stories 10417535 Wed Mar 20 21:34:12.793 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._8 Wed Mar 20 21:34:12.816 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._9 Wed Mar 20 21:35:08.457 [rsSync] 10665968 objects cloned so far from collection newsblur.stories Wed Mar 20 21:35:09.699 [rsSync] clone newsblur.stories 10667775 Wed Mar 20 21:36:12.314 [rsSync] clone newsblur.stories 10850047 Wed Mar 20 21:36:12.317 [rsSync] 10850118 objects cloned so far from collection newsblur.stories Wed Mar 20 21:37:16.167 [rsSync] clone newsblur.stories 11000063 Wed Mar 20 21:37:16.168 [rsSync] 11000105 objects cloned so far from collection newsblur.stories Wed Mar 20 21:38:16.005 [rsSync] clone newsblur.stories 11151231 Wed Mar 20 21:38:17.005 [rsSync] 11155553 objects cloned so far from collection newsblur.stories Wed Mar 20 21:39:19.158 [rsSync] 11319256 objects cloned so far from collection newsblur.stories Wed Mar 20 21:39:19.160 [rsSync] clone newsblur.stories 11319295 Wed Mar 20 21:39:41.921 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.14, filling with zeroes... Wed Mar 20 21:39:41.927 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.14, size: 2047MB, took 0.006 secs Wed Mar 20 21:40:19.693 [rsSync] clone newsblur.stories 11405183 Wed Mar 20 21:40:22.574 [rsSync] 11408918 objects cloned so far from collection newsblur.stories Wed Mar 20 21:41:19.497 [rsSync] clone newsblur.stories 11564543 Wed Mar 20 21:41:26.143 [rsSync] 11573256 objects cloned so far from collection newsblur.stories Wed Mar 20 21:42:19.082 [rsSync] clone newsblur.stories 11742847 Wed Mar 20 21:42:28.260 [rsSync] 11760708 objects cloned so far from collection newsblur.stories Wed Mar 20 21:43:19.005 [rsSync] clone newsblur.stories 11926911 Wed Mar 20 21:43:29.006 [rsSync] 11944604 objects cloned so far from collection newsblur.stories Wed Mar 20 21:44:19.150 [rsSync] clone newsblur.stories 12159103 Wed Mar 20 21:44:30.165 [rsSync] 12183491 objects cloned so far from collection newsblur.stories Wed Mar 20 21:45:16.249 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._10 Wed Mar 20 21:45:16.271 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._11 Wed Mar 20 21:45:25.378 [rsSync] clone newsblur.stories 12400511 Wed Mar 20 21:45:34.308 [rsSync] 12407265 objects cloned so far from collection newsblur.stories Wed Mar 20 21:46:26.327 [rsSync] clone newsblur.stories 12510591 Wed Mar 20 21:46:35.061 [rsSync] 12532614 objects cloned so far from collection newsblur.stories Wed Mar 20 21:47:27.184 [rsSync] clone newsblur.stories 12714367 Wed Mar 20 21:47:36.036 [rsSync] 12761087 objects cloned so far from collection newsblur.stories Wed Mar 20 21:48:28.172 [rsSync] clone newsblur.stories 12801663 Wed Mar 20 21:48:39.094 [rsSync] 12816967 objects cloned so far from collection newsblur.stories Wed Mar 20 21:48:48.115 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.15, filling with zeroes... Wed Mar 20 21:48:48.122 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.15, size: 2047MB, took 0.006 secs Wed Mar 20 21:49:28.043 [rsSync] clone newsblur.stories 12937471 Wed Mar 20 21:49:40.446 [rsSync] 12991338 objects cloned so far from collection newsblur.stories Wed Mar 20 21:50:30.314 [rsSync] clone newsblur.stories 13138943 Wed Mar 20 21:50:41.661 [rsSync] 13173710 objects cloned so far from collection newsblur.stories Wed Mar 20 21:51:30.566 [rsSync] clone newsblur.stories 13351295 Wed Mar 20 21:51:42.359 [rsSync] 13373100 objects cloned so far from collection newsblur.stories Wed Mar 20 21:52:30.796 [rsSync] clone newsblur.stories 13577599 Wed Mar 20 21:52:43.205 [rsSync] 13636879 objects cloned so far from collection newsblur.stories Wed Mar 20 21:53:30.536 [rsSync] clone newsblur.stories 13788287 Wed Mar 20 21:53:44.343 [rsSync] 13835888 objects cloned so far from collection newsblur.stories Wed Mar 20 21:54:30.405 [rsSync] clone newsblur.stories 13976319 Wed Mar 20 21:54:45.299 [rsSync] 14031849 objects cloned so far from collection newsblur.stories Wed Mar 20 21:55:06.340 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._12 Wed Mar 20 21:55:32.552 [rsSync] clone newsblur.stories 14185599 Wed Mar 20 21:55:47.995 [rsSync] 14192377 objects cloned so far from collection newsblur.stories Wed Mar 20 21:56:30.101 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.16, filling with zeroes... Wed Mar 20 21:56:30.107 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.16, size: 2047MB, took 0.006 secs Wed Mar 20 21:56:33.377 [rsSync] clone newsblur.stories 14267519 Wed Mar 20 21:56:48.333 [rsSync] 14295755 objects cloned so far from collection newsblur.stories Wed Mar 20 21:57:33.496 [rsSync] clone newsblur.stories 14475263 Wed Mar 20 21:57:49.717 [rsSync] 14519529 objects cloned so far from collection newsblur.stories Wed Mar 20 21:58:33.668 [rsSync] clone newsblur.stories 14657791 Wed Mar 20 21:58:50.599 [rsSync] 14705965 objects cloned so far from collection newsblur.stories Wed Mar 20 21:59:35.348 [rsSync] clone newsblur.stories 14830719 Wed Mar 20 21:59:51.512 [rsSync] 14849602 objects cloned so far from collection newsblur.stories Wed Mar 20 22:00:27.783 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._13 Wed Mar 20 22:00:36.738 [rsSync] clone newsblur.stories 14970879 Wed Mar 20 22:00:54.413 [rsSync] 15000097 objects cloned so far from collection newsblur.stories Wed Mar 20 22:01:36.139 [rsSync] clone newsblur.stories 15155711 Wed Mar 20 22:01:55.055 [rsSync] 15226538 objects cloned so far from collection newsblur.stories Wed Mar 20 22:02:36.007 [rsSync] clone newsblur.stories 15340799 Wed Mar 20 22:02:57.452 [rsSync] 15411196 objects cloned so far from collection newsblur.stories Wed Mar 20 22:03:37.827 [rsSync] clone newsblur.stories 15438591 Wed Mar 20 22:03:59.055 [rsSync] 15469235 objects cloned so far from collection newsblur.stories Wed Mar 20 22:04:37.461 [rsSync] clone newsblur.stories 15566079 Wed Mar 20 22:04:37.513 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.17, filling with zeroes... Wed Mar 20 22:04:37.520 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.17, size: 2047MB, took 0.006 secs Wed Mar 20 22:05:00.021 [rsSync] 15632049 objects cloned so far from collection newsblur.stories Wed Mar 20 22:05:37.160 [rsSync] clone newsblur.stories 15714687 Wed Mar 20 22:05:53.331 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._14 Wed Mar 20 22:06:01.528 [rsSync] 15802229 objects cloned so far from collection newsblur.stories Wed Mar 20 22:06:37.097 [rsSync] clone newsblur.stories 15881599 Wed Mar 20 22:07:02.227 [rsSync] 15979267 objects cloned so far from collection newsblur.stories Wed Mar 20 22:07:37.826 [rsSync] clone newsblur.stories 16050047 Wed Mar 20 22:08:03.427 [rsSync] 16116427 objects cloned so far from collection newsblur.stories Wed Mar 20 22:08:37.667 [rsSync] clone newsblur.stories 16238591 Wed Mar 20 22:09:04.530 [rsSync] 16305022 objects cloned so far from collection newsblur.stories Wed Mar 20 22:09:37.005 [rsSync] clone newsblur.stories 16397695 Wed Mar 20 22:10:05.279 [rsSync] 16530320 objects cloned so far from collection newsblur.stories Wed Mar 20 22:10:12.054 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._15 Wed Mar 20 22:10:12.080 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._16 Wed Mar 20 22:10:39.187 [rsSync] clone newsblur.stories 16574719 Wed Mar 20 22:11:09.085 [rsSync] 16583279 objects cloned so far from collection newsblur.stories Wed Mar 20 22:11:42.522 [rsSync] clone newsblur.stories 16596095 Wed Mar 20 22:12:12.720 [rsSync] 16617188 objects cloned so far from collection newsblur.stories Wed Mar 20 22:12:45.117 [rsSync] clone newsblur.stories 16641407 Wed Mar 20 22:13:14.391 [rsSync] 16686149 objects cloned so far from collection newsblur.stories Wed Mar 20 22:13:45.150 [rsSync] clone newsblur.stories 16729471 Wed Mar 20 22:14:15.038 [rsSync] 16845026 objects cloned so far from collection newsblur.stories Wed Mar 20 22:14:16.037 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.18, filling with zeroes... Wed Mar 20 22:14:16.045 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.18, size: 2047MB, took 0.008 secs Wed Mar 20 22:14:46.074 [rsSync] clone newsblur.stories 16939775 Wed Mar 20 22:15:16.092 [rsSync] 17034256 objects cloned so far from collection newsblur.stories Wed Mar 20 22:15:46.006 [rsSync] clone newsblur.stories 17182591 Wed Mar 20 22:16:17.338 [rsSync] 17285716 objects cloned so far from collection newsblur.stories Wed Mar 20 22:16:46.326 [rsSync] clone newsblur.stories 17385215 Wed Mar 20 22:17:18.086 [rsSync] 17534763 objects cloned so far from collection newsblur.stories Wed Mar 20 22:17:47.140 [rsSync] clone newsblur.stories 17646591 Wed Mar 20 22:18:19.929 [rsSync] 17760823 objects cloned so far from collection newsblur.stories Wed Mar 20 22:18:47.180 [rsSync] clone newsblur.stories 17827455 Wed Mar 20 22:19:20.196 [rsSync] 17890490 objects cloned so far from collection newsblur.stories Wed Mar 20 22:19:50.222 [rsSync] clone newsblur.stories 17934975 Wed Mar 20 22:20:12.161 [conn20446] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 4447, after repl: 4447, at end: 4447 } Wed Mar 20 22:20:12.162 [conn20446] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 reslen:3446 4519ms Wed Mar 20 22:20:24.590 [rsSync] 18043525 objects cloned so far from collection newsblur.stories Wed Mar 20 22:20:50.135 [rsSync] clone newsblur.stories 18139903 Wed Mar 20 22:21:17.061 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.19, filling with zeroes... Wed Mar 20 22:21:17.068 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.19, size: 2047MB, took 0.006 secs Wed Mar 20 22:21:25.863 [rsSync] 18240883 objects cloned so far from collection newsblur.stories Wed Mar 20 22:21:40.866 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._17 Wed Mar 20 22:21:52.299 [rsSync] clone newsblur.stories 18291455 Wed Mar 20 22:22:26.599 [rsSync] 18373344 objects cloned so far from collection newsblur.stories Wed Mar 20 22:22:52.776 [rsSync] clone newsblur.stories 18453119 Wed Mar 20 22:23:28.081 [rsSync] 18576036 objects cloned so far from collection newsblur.stories Wed Mar 20 22:23:52.357 [rsSync] clone newsblur.stories 18652799 Wed Mar 20 22:24:29.767 [rsSync] 18757011 objects cloned so far from collection newsblur.stories Wed Mar 20 22:24:53.270 [rsSync] clone newsblur.stories 18822271 Wed Mar 20 22:25:30.284 [rsSync] 18957925 objects cloned so far from collection newsblur.stories Wed Mar 20 22:25:53.380 [rsSync] clone newsblur.stories 19010559 Wed Mar 20 22:26:33.140 [rsSync] 19041872 objects cloned so far from collection newsblur.stories Wed Mar 20 22:26:53.217 [rsSync] clone newsblur.stories 19058303 Wed Mar 20 22:27:38.324 [rsSync] 19084544 objects cloned so far from collection newsblur.stories Wed Mar 20 22:27:53.947 [rsSync] clone newsblur.stories 19104639 Wed Mar 20 22:28:14.770 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._18 Wed Mar 20 22:28:14.793 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._19 Wed Mar 20 22:28:40.895 [rsSync] 19181953 objects cloned so far from collection newsblur.stories Wed Mar 20 22:28:53.077 [rsSync] clone newsblur.stories 19212927 Wed Mar 20 22:29:41.010 [rsSync] 19318097 objects cloned so far from collection newsblur.stories Wed Mar 20 22:29:54.544 [rsSync] clone newsblur.stories 19343999 Wed Mar 20 22:30:42.032 [rsSync] 19460845 objects cloned so far from collection newsblur.stories Wed Mar 20 22:30:55.413 [rsSync] clone newsblur.stories 19486591 Wed Mar 20 22:31:43.591 [rsSync] 19583908 objects cloned so far from collection newsblur.stories Wed Mar 20 22:31:52.646 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.20, filling with zeroes... Wed Mar 20 22:31:52.653 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.20, size: 2047MB, took 0.006 secs Wed Mar 20 22:31:55.335 [rsSync] clone newsblur.stories 19630335 Wed Mar 20 22:32:45.418 [rsSync] 19721449 objects cloned so far from collection newsblur.stories Wed Mar 20 22:32:56.293 [rsSync] clone newsblur.stories 19739263 Wed Mar 20 22:33:47.129 [rsSync] 19833209 objects cloned so far from collection newsblur.stories Wed Mar 20 22:33:56.224 [rsSync] clone newsblur.stories 19849727 Wed Mar 20 22:34:48.028 [rsSync] 19984974 objects cloned so far from collection newsblur.stories Wed Mar 20 22:34:57.611 [rsSync] clone newsblur.stories 20000127 Wed Mar 20 22:35:49.081 [rsSync] 20096099 objects cloned so far from collection newsblur.stories Wed Mar 20 22:35:57.929 [rsSync] clone newsblur.stories 20137855 Wed Mar 20 22:36:51.404 [rsSync] 20248753 objects cloned so far from collection newsblur.stories Wed Mar 20 22:36:58.299 [rsSync] clone newsblur.stories 20254591 Wed Mar 20 22:37:53.895 [rsSync] 20287742 objects cloned so far from collection newsblur.stories Wed Mar 20 22:37:58.651 [rsSync] clone newsblur.stories 20290687 Wed Mar 20 22:38:54.084 [rsSync] 20348321 objects cloned so far from collection newsblur.stories Wed Mar 20 22:38:59.156 [rsSync] clone newsblur.stories 20359935 Wed Mar 20 22:39:55.080 [rsSync] 20483322 objects cloned so far from collection newsblur.stories Wed Mar 20 22:40:00.145 [rsSync] clone newsblur.stories 20493695 Wed Mar 20 22:40:56.173 [rsSync] 20659979 objects cloned so far from collection newsblur.stories Wed Mar 20 22:41:00.898 [rsSync] clone newsblur.stories 20674815 Wed Mar 20 22:41:57.192 [rsSync] 20831810 objects cloned so far from collection newsblur.stories Wed Mar 20 22:42:00.435 [rsSync] clone newsblur.stories 20841215 Wed Mar 20 22:43:05.352 [rsSync] clone newsblur.stories 21000063 Wed Mar 20 22:43:05.352 [rsSync] 21000085 objects cloned so far from collection newsblur.stories Wed Mar 20 22:44:05.830 [rsSync] clone newsblur.stories 21116671 Wed Mar 20 22:44:06.761 [rsSync] 21117814 objects cloned so far from collection newsblur.stories Wed Mar 20 22:44:28.204 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._20 Wed Mar 20 22:44:35.605 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.21, filling with zeroes... Wed Mar 20 22:44:35.612 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.21, size: 2047MB, took 0.006 secs Wed Mar 20 22:45:05.878 [rsSync] clone newsblur.stories 21244543 Wed Mar 20 22:45:07.147 [rsSync] 21247608 objects cloned so far from collection newsblur.stories Wed Mar 20 22:45:08.991 [conn24129] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:52 reslen:3106 105ms Wed Mar 20 22:46:06.184 [rsSync] clone newsblur.stories 21358079 Wed Mar 20 22:46:08.418 [rsSync] 21361654 objects cloned so far from collection newsblur.stories Wed Mar 20 22:47:08.831 [rsSync] clone newsblur.stories 21433855 Wed Mar 20 22:47:10.838 [rsSync] 21435314 objects cloned so far from collection newsblur.stories Wed Mar 20 22:48:08.790 [rsSync] clone newsblur.stories 21459455 Wed Mar 20 22:48:13.003 [rsSync] 21461349 objects cloned so far from collection newsblur.stories Wed Mar 20 22:49:14.466 [rsSync] 21493607 objects cloned so far from collection newsblur.stories Wed Mar 20 22:49:14.468 [rsSync] clone newsblur.stories 21493631 Wed Mar 20 22:50:15.154 [rsSync] clone newsblur.stories 21532799 Wed Mar 20 22:50:15.156 [rsSync] 21532850 objects cloned so far from collection newsblur.stories Wed Mar 20 22:51:15.173 [rsSync] clone newsblur.stories 21597823 Wed Mar 20 22:51:18.805 [rsSync] 21602700 objects cloned so far from collection newsblur.stories Wed Mar 20 22:52:15.751 [rsSync] clone newsblur.stories 21766527 Wed Mar 20 22:52:19.340 [rsSync] 21772245 objects cloned so far from collection newsblur.stories Wed Mar 20 22:53:15.456 [rsSync] clone newsblur.stories 21871103 Wed Mar 20 22:53:20.326 [rsSync] 21873337 objects cloned so far from collection newsblur.stories Wed Mar 20 22:53:51.318 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._21 Wed Mar 20 22:54:15.535 [rsSync] clone newsblur.stories 21970815 Wed Mar 20 22:54:21.141 [rsSync] 21996781 objects cloned so far from collection newsblur.stories Wed Mar 20 22:55:15.235 [rsSync] clone newsblur.stories 22077951 Wed Mar 20 22:55:26.546 [rsSync] 22099651 objects cloned so far from collection newsblur.stories Wed Mar 20 22:56:15.103 [rsSync] clone newsblur.stories 22249471 Wed Mar 20 22:56:27.046 [rsSync] 22271228 objects cloned so far from collection newsblur.stories Wed Mar 20 22:56:28.472 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.22, filling with zeroes... Wed Mar 20 22:56:28.479 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.22, size: 2047MB, took 0.007 secs Wed Mar 20 22:57:17.113 [rsSync] clone newsblur.stories 22342271 Wed Mar 20 22:57:28.041 [rsSync] 22349968 objects cloned so far from collection newsblur.stories Wed Mar 20 22:58:17.752 [rsSync] clone newsblur.stories 22395647 Wed Mar 20 22:58:29.006 [rsSync] 22415246 objects cloned so far from collection newsblur.stories Wed Mar 20 22:59:17.564 [rsSync] clone newsblur.stories 22505855 Wed Mar 20 22:59:30.451 [rsSync] 22562312 objects cloned so far from collection newsblur.stories Wed Mar 20 22:59:33.670 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._22 Wed Mar 20 23:00:17.006 [rsSync] clone newsblur.stories 22654719 Wed Mar 20 23:00:31.012 [rsSync] 22698710 objects cloned so far from collection newsblur.stories Wed Mar 20 23:01:17.140 [rsSync] clone newsblur.stories 22796287 Wed Mar 20 23:01:33.017 [rsSync] 22818598 objects cloned so far from collection newsblur.stories Wed Mar 20 23:02:18.386 [rsSync] clone newsblur.stories 22923391 Wed Mar 20 23:02:34.297 [rsSync] 22945471 objects cloned so far from collection newsblur.stories Wed Mar 20 23:03:18.280 [rsSync] clone newsblur.stories 23032703 Wed Mar 20 23:03:35.854 [rsSync] 23050627 objects cloned so far from collection newsblur.stories Wed Mar 20 23:04:18.062 [rsSync] clone newsblur.stories 23146111 Wed Mar 20 23:04:36.137 [rsSync] 23175468 objects cloned so far from collection newsblur.stories Wed Mar 20 23:05:06.242 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._23 Wed Mar 20 23:05:18.099 [rsSync] clone newsblur.stories 23270143 Wed Mar 20 23:05:38.253 [rsSync] 23297769 objects cloned so far from collection newsblur.stories Wed Mar 20 23:06:18.651 [rsSync] clone newsblur.stories 23320831 Wed Mar 20 23:06:40.062 [rsSync] 23329265 objects cloned so far from collection newsblur.stories Wed Mar 20 23:07:20.091 [rsSync] clone newsblur.stories 23369343 Wed Mar 20 23:07:36.169 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.23, filling with zeroes... Wed Mar 20 23:07:36.177 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.23, size: 2047MB, took 0.006 secs Wed Mar 20 23:07:41.584 [rsSync] 23400004 objects cloned so far from collection newsblur.stories Wed Mar 20 23:08:20.786 [rsSync] clone newsblur.stories 23526271 Wed Mar 20 23:08:42.835 [rsSync] 23564469 objects cloned so far from collection newsblur.stories Wed Mar 20 23:09:21.239 [rsSync] clone newsblur.stories 23637759 Wed Mar 20 23:09:43.993 [rsSync] 23670641 objects cloned so far from collection newsblur.stories Wed Mar 20 23:10:26.060 [rsSync] clone newsblur.stories 23752447 Wed Mar 20 23:10:47.026 [rsSync] 23759541 objects cloned so far from collection newsblur.stories Wed Mar 20 23:11:26.549 [rsSync] clone newsblur.stories 23825023 Wed Mar 20 23:11:48.676 [rsSync] 23881334 objects cloned so far from collection newsblur.stories Wed Mar 20 23:12:26.173 [rsSync] clone newsblur.stories 23936767 Wed Mar 20 23:12:40.813 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._24 Wed Mar 20 23:12:49.255 [rsSync] 23961471 objects cloned so far from collection newsblur.stories Wed Mar 20 23:13:26.721 [rsSync] clone newsblur.stories 24029567 Wed Mar 20 23:13:50.201 [rsSync] 24089741 objects cloned so far from collection newsblur.stories Wed Mar 20 23:14:27.138 [rsSync] clone newsblur.stories 24126079 Wed Mar 20 23:14:51.017 [rsSync] 24163655 objects cloned so far from collection newsblur.stories Wed Mar 20 23:15:30.974 [rsSync] clone newsblur.stories 24198527 Wed Mar 20 23:15:52.720 [rsSync] 24207597 objects cloned so far from collection newsblur.stories Wed Mar 20 23:16:31.120 [rsSync] clone newsblur.stories 24219135 Wed Mar 20 23:16:53.818 [rsSync] 24232235 objects cloned so far from collection newsblur.stories Wed Mar 20 23:17:34.108 [rsSync] clone newsblur.stories 24252031 Wed Mar 20 23:17:55.207 [rsSync] 24265509 objects cloned so far from collection newsblur.stories Wed Mar 20 23:18:37.000 [rsSync] clone newsblur.stories 24305791 Wed Mar 20 23:18:58.934 [rsSync] 24332438 objects cloned so far from collection newsblur.stories Wed Mar 20 23:19:40.664 [rsSync] clone newsblur.stories 24403839 Wed Mar 20 23:19:59.220 [rsSync] 24451818 objects cloned so far from collection newsblur.stories Wed Mar 20 23:20:40.604 [rsSync] clone newsblur.stories 24542591 Wed Mar 20 23:21:00.682 [rsSync] 24570182 objects cloned so far from collection newsblur.stories Wed Mar 20 23:21:10.523 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.24, filling with zeroes... Wed Mar 20 23:21:10.533 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.24, size: 2047MB, took 0.009 secs Wed Mar 20 23:21:41.596 [rsSync] clone newsblur.stories 24617215 Wed Mar 20 23:22:02.272 [rsSync] 24658701 objects cloned so far from collection newsblur.stories Wed Mar 20 23:22:02.469 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._25 Wed Mar 20 23:22:42.131 [rsSync] clone newsblur.stories 24730367 Wed Mar 20 23:23:04.592 [rsSync] 24760555 objects cloned so far from collection newsblur.stories Wed Mar 20 23:23:42.211 [rsSync] clone newsblur.stories 24789247 Wed Mar 20 23:24:05.153 [rsSync] 24817578 objects cloned so far from collection newsblur.stories Wed Mar 20 23:24:42.355 [rsSync] clone newsblur.stories 24847615 Wed Mar 20 23:25:06.554 [rsSync] 24873712 objects cloned so far from collection newsblur.stories Wed Mar 20 23:25:42.275 [rsSync] clone newsblur.stories 24931071 Wed Mar 20 23:26:10.226 [rsSync] 24981154 objects cloned so far from collection newsblur.stories Wed Mar 20 23:26:46.880 [rsSync] clone newsblur.stories 25000063 Wed Mar 20 23:27:11.361 [rsSync] 25020905 objects cloned so far from collection newsblur.stories Wed Mar 20 23:27:46.876 [rsSync] clone newsblur.stories 25061119 Wed Mar 20 23:28:12.528 [rsSync] 25090374 objects cloned so far from collection newsblur.stories Wed Mar 20 23:28:46.979 [rsSync] clone newsblur.stories 25145215 Wed Mar 20 23:29:14.099 [rsSync] 25193625 objects cloned so far from collection newsblur.stories Wed Mar 20 23:29:47.220 [rsSync] clone newsblur.stories 25271423 Wed Mar 20 23:30:18.029 [rsSync] 25298146 objects cloned so far from collection newsblur.stories Wed Mar 20 23:30:20.057 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._26 Wed Mar 20 23:30:47.892 [rsSync] clone newsblur.stories 25323391 Wed Mar 20 23:31:20.182 [rsSync] 25351867 objects cloned so far from collection newsblur.stories Wed Mar 20 23:31:47.479 [rsSync] clone newsblur.stories 25371135 Wed Mar 20 23:32:21.008 [rsSync] 25413208 objects cloned so far from collection newsblur.stories Wed Mar 20 23:32:48.644 [rsSync] clone newsblur.stories 25442431 Wed Mar 20 23:33:23.324 [rsSync] 25480899 objects cloned so far from collection newsblur.stories Wed Mar 20 23:33:50.079 [rsSync] clone newsblur.stories 25515903 Wed Mar 20 23:34:25.974 [rsSync] 25550749 objects cloned so far from collection newsblur.stories Wed Mar 20 23:34:40.239 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.25, filling with zeroes... Wed Mar 20 23:34:40.246 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.25, size: 2047MB, took 0.007 secs Wed Mar 20 23:34:53.726 [rsSync] clone newsblur.stories 25559551 Wed Mar 20 23:35:26.740 [rsSync] 25593675 objects cloned so far from collection newsblur.stories Wed Mar 20 23:35:53.139 [rsSync] clone newsblur.stories 25613055 Wed Mar 20 23:36:27.142 [rsSync] 25636347 objects cloned so far from collection newsblur.stories Wed Mar 20 23:36:53.511 [rsSync] clone newsblur.stories 25686143 Wed Mar 20 23:37:28.324 [rsSync] 25706959 objects cloned so far from collection newsblur.stories Wed Mar 20 23:37:54.286 [rsSync] clone newsblur.stories 25717375 Wed Mar 20 23:38:31.930 [rsSync] 25733883 objects cloned so far from collection newsblur.stories Wed Mar 20 23:38:54.016 [rsSync] clone newsblur.stories 25749759 Wed Mar 20 23:39:10.948 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._27 Wed Mar 20 23:39:10.971 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._28 Wed Mar 20 23:39:33.174 [rsSync] 25772618 objects cloned so far from collection newsblur.stories Wed Mar 20 23:39:55.173 [rsSync] clone newsblur.stories 25797503 Wed Mar 20 23:40:40.666 [rsSync] 25827736 objects cloned so far from collection newsblur.stories Wed Mar 20 23:40:55.103 [rsSync] clone newsblur.stories 25846143 Wed Mar 20 23:41:43.170 [rsSync] 25902920 objects cloned so far from collection newsblur.stories Wed Mar 20 23:41:55.148 [rsSync] clone newsblur.stories 25928703 Wed Mar 20 23:42:44.157 [rsSync] 26037413 objects cloned so far from collection newsblur.stories Wed Mar 20 23:42:55.094 [rsSync] clone newsblur.stories 26059519 Wed Mar 20 23:43:45.007 [rsSync] 26118058 objects cloned so far from collection newsblur.stories Wed Mar 20 23:43:55.100 [rsSync] clone newsblur.stories 26138367 Wed Mar 20 23:44:46.010 [rsSync] 26289000 objects cloned so far from collection newsblur.stories Wed Mar 20 23:44:55.101 [rsSync] clone newsblur.stories 26308863 Wed Mar 20 23:45:47.337 [rsSync] 26382472 objects cloned so far from collection newsblur.stories Wed Mar 20 23:45:47.373 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.26, filling with zeroes... Wed Mar 20 23:45:47.382 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.26, size: 2047MB, took 0.008 secs Wed Mar 20 23:45:55.748 [rsSync] clone newsblur.stories 26388479 Wed Mar 20 23:46:49.410 [rsSync] 26445337 objects cloned so far from collection newsblur.stories Wed Mar 20 23:46:55.151 [rsSync] clone newsblur.stories 26449791 Wed Mar 20 23:47:50.543 [rsSync] 26493724 objects cloned so far from collection newsblur.stories Wed Mar 20 23:47:55.391 [rsSync] clone newsblur.stories 26497791 Wed Mar 20 23:48:52.424 [rsSync] 26672794 objects cloned so far from collection newsblur.stories Wed Mar 20 23:48:55.141 [rsSync] clone newsblur.stories 26681727 Wed Mar 20 23:49:53.414 [rsSync] 26825575 objects cloned so far from collection newsblur.stories Wed Mar 20 23:49:55.280 [rsSync] clone newsblur.stories 26828671 Wed Mar 20 23:50:05.703 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._29 Wed Mar 20 23:50:55.214 [rsSync] clone newsblur.stories 26945663 Wed Mar 20 23:50:55.216 [rsSync] 26945717 objects cloned so far from collection newsblur.stories Wed Mar 20 23:51:58.105 [rsSync] clone newsblur.stories 27000063 Wed Mar 20 23:51:58.118 [rsSync] 27000073 objects cloned so far from collection newsblur.stories Wed Mar 20 23:52:59.619 [rsSync] clone newsblur.stories 27059839 Wed Mar 20 23:52:59.621 [rsSync] 27059890 objects cloned so far from collection newsblur.stories Wed Mar 20 23:54:00.732 [rsSync] 27118310 objects cloned so far from collection newsblur.stories Wed Mar 20 23:54:00.733 [rsSync] clone newsblur.stories 27118335 Wed Mar 20 23:55:00.062 [rsSync] clone newsblur.stories 27274879 Wed Mar 20 23:55:01.407 [rsSync] 27284172 objects cloned so far from collection newsblur.stories Wed Mar 20 23:55:16.916 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.27, filling with zeroes... Wed Mar 20 23:55:16.923 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.27, size: 2047MB, took 0.007 secs Wed Mar 20 23:55:40.813 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._30 Wed Mar 20 23:56:00.039 [rsSync] clone newsblur.stories 27375743 Wed Mar 20 23:56:02.248 [rsSync] 27378279 objects cloned so far from collection newsblur.stories Wed Mar 20 23:57:00.653 [rsSync] clone newsblur.stories 27448447 Wed Mar 20 23:57:03.038 [rsSync] 27451050 objects cloned so far from collection newsblur.stories Wed Mar 20 23:58:00.723 [rsSync] clone newsblur.stories 27497599 Wed Mar 20 23:58:04.874 [rsSync] 27501088 objects cloned so far from collection newsblur.stories Wed Mar 20 23:59:01.114 [rsSync] clone newsblur.stories 27548671 Wed Mar 20 23:59:05.029 [rsSync] 27550491 objects cloned so far from collection newsblur.stories Thu Mar 21 00:00:01.634 [rsSync] clone newsblur.stories 27591423 Thu Mar 21 00:00:06.463 [rsSync] 27593925 objects cloned so far from collection newsblur.stories Thu Mar 21 00:00:52.164 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._31 Thu Mar 21 00:01:02.046 [rsSync] clone newsblur.stories 27643519 Thu Mar 21 00:01:07.076 [rsSync] 27647392 objects cloned so far from collection newsblur.stories Thu Mar 21 00:02:02.151 [rsSync] clone newsblur.stories 27748991 Thu Mar 21 00:02:08.602 [rsSync] 27768423 objects cloned so far from collection newsblur.stories Thu Mar 21 00:03:02.349 [rsSync] clone newsblur.stories 27812095 Thu Mar 21 00:03:09.209 [rsSync] 27820112 objects cloned so far from collection newsblur.stories Thu Mar 21 00:04:02.577 [rsSync] clone newsblur.stories 27861631 Thu Mar 21 00:04:11.299 [rsSync] 27866848 objects cloned so far from collection newsblur.stories Thu Mar 21 00:04:13.214 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.28, filling with zeroes... Thu Mar 21 00:04:13.221 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.28, size: 2047MB, took 0.006 secs Thu Mar 21 00:05:02.521 [rsSync] clone newsblur.stories 27898495 Thu Mar 21 00:05:12.229 [rsSync] 27908631 objects cloned so far from collection newsblur.stories Thu Mar 21 00:06:02.746 [rsSync] clone newsblur.stories 27964415 Thu Mar 21 00:06:14.089 [rsSync] 27978227 objects cloned so far from collection newsblur.stories Thu Mar 21 00:06:42.484 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._32 Thu Mar 21 00:07:02.178 [rsSync] clone newsblur.stories 28057087 Thu Mar 21 00:07:15.006 [rsSync] 28091765 objects cloned so far from collection newsblur.stories Thu Mar 21 00:08:02.045 [rsSync] clone newsblur.stories 28286591 Thu Mar 21 00:08:18.126 [rsSync] 28301315 objects cloned so far from collection newsblur.stories Thu Mar 21 00:09:03.351 [rsSync] clone newsblur.stories 28331775 Thu Mar 21 00:09:20.714 [rsSync] 28349321 objects cloned so far from collection newsblur.stories Thu Mar 21 00:10:06.078 [rsSync] clone newsblur.stories 28386303 Thu Mar 21 00:10:21.398 [rsSync] 28401899 objects cloned so far from collection newsblur.stories Thu Mar 21 00:11:06.255 [rsSync] clone newsblur.stories 28456447 Thu Mar 21 00:11:23.428 [rsSync] 28479496 objects cloned so far from collection newsblur.stories Thu Mar 21 00:11:42.962 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._33 Thu Mar 21 00:12:06.140 [rsSync] clone newsblur.stories 28601599 Thu Mar 21 00:12:24.147 [rsSync] 28675330 objects cloned so far from collection newsblur.stories Thu Mar 21 00:12:31.702 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.29, filling with zeroes... Thu Mar 21 00:12:31.710 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.29, size: 2047MB, took 0.007 secs Thu Mar 21 00:13:06.007 [rsSync] clone newsblur.stories 28901631 Thu Mar 21 00:13:26.264 [rsSync] 28921075 objects cloned so far from collection newsblur.stories Thu Mar 21 00:14:06.021 [rsSync] clone newsblur.stories 28972159 Thu Mar 21 00:14:30.430 [rsSync] 29000069 objects cloned so far from collection newsblur.stories Thu Mar 21 00:15:06.811 [rsSync] clone newsblur.stories 29053439 Thu Mar 21 00:15:28.287 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._34 Thu Mar 21 00:15:31.293 [rsSync] 29086302 objects cloned so far from collection newsblur.stories Thu Mar 21 00:16:06.377 [rsSync] clone newsblur.stories 29122303 Thu Mar 21 00:16:32.117 [rsSync] 29160216 objects cloned so far from collection newsblur.stories Thu Mar 21 00:17:06.102 [rsSync] clone newsblur.stories 29179391 Thu Mar 21 00:17:33.715 [rsSync] 29204285 objects cloned so far from collection newsblur.stories Thu Mar 21 00:18:07.181 [rsSync] clone newsblur.stories 29256447 Thu Mar 21 00:18:35.197 [rsSync] 29304615 objects cloned so far from collection newsblur.stories Thu Mar 21 00:19:07.158 [rsSync] clone newsblur.stories 29371007 Thu Mar 21 00:19:37.069 [rsSync] 29438346 objects cloned so far from collection newsblur.stories Thu Mar 21 00:20:09.240 [rsSync] clone newsblur.stories 29526271 Thu Mar 21 00:20:36.578 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.30, filling with zeroes... Thu Mar 21 00:20:36.586 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.30, size: 2047MB, took 0.007 secs Thu Mar 21 00:20:38.407 [rsSync] 29633164 objects cloned so far from collection newsblur.stories Thu Mar 21 00:20:50.017 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._35 Thu Mar 21 00:21:09.229 [rsSync] clone newsblur.stories 29668095 Thu Mar 21 00:21:42.095 [rsSync] 29689298 objects cloned so far from collection newsblur.stories Thu Mar 21 00:22:09.069 [rsSync] clone newsblur.stories 29705727 Thu Mar 21 00:22:43.706 [rsSync] 29752798 objects cloned so far from collection newsblur.stories Thu Mar 21 00:23:10.196 [rsSync] clone newsblur.stories 29783679 Thu Mar 21 00:23:44.316 [rsSync] 29830522 objects cloned so far from collection newsblur.stories Thu Mar 21 00:24:10.013 [rsSync] clone newsblur.stories 29916415 Thu Mar 21 00:24:45.813 [rsSync] 30062043 objects cloned so far from collection newsblur.stories Thu Mar 21 00:24:53.469 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._36 Thu Mar 21 00:25:10.354 [rsSync] clone newsblur.stories 30140799 Thu Mar 21 00:25:47.274 [rsSync] 30276038 objects cloned so far from collection newsblur.stories Thu Mar 21 00:26:10.618 [rsSync] clone newsblur.stories 30322815 Thu Mar 21 00:26:48.582 [rsSync] 30410277 objects cloned so far from collection newsblur.stories Thu Mar 21 00:27:10.001 [rsSync] clone newsblur.stories 30465663 Thu Mar 21 00:27:44.210 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.31, filling with zeroes... Thu Mar 21 00:27:44.216 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.31, size: 2047MB, took 0.005 secs Thu Mar 21 00:27:49.325 [rsSync] 30623256 objects cloned so far from collection newsblur.stories Thu Mar 21 00:28:10.725 [rsSync] clone newsblur.stories 30719999 Thu Mar 21 00:28:45.875 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._37 Thu Mar 21 00:28:51.273 [rsSync] 30899608 objects cloned so far from collection newsblur.stories Thu Mar 21 00:29:10.961 [rsSync] clone newsblur.stories 30983039 Thu Mar 21 00:29:52.209 [rsSync] 31123128 objects cloned so far from collection newsblur.stories Thu Mar 21 00:30:10.003 [rsSync] clone newsblur.stories 31180543 Thu Mar 21 00:30:53.091 [rsSync] 31312104 objects cloned so far from collection newsblur.stories Thu Mar 21 00:31:11.496 [rsSync] clone newsblur.stories 31348223 Thu Mar 21 00:31:54.276 [rsSync] 31429579 objects cloned so far from collection newsblur.stories Thu Mar 21 00:32:11.586 [rsSync] clone newsblur.stories 31469695 Thu Mar 21 00:32:33.594 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._38 Thu Mar 21 00:32:56.731 [rsSync] 31580836 objects cloned so far from collection newsblur.stories Thu Mar 21 00:33:13.263 [rsSync] clone newsblur.stories 31612671 Thu Mar 21 00:33:30.765 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.32, filling with zeroes... Thu Mar 21 00:33:30.772 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.32, size: 2047MB, took 0.006 secs Thu Mar 21 00:33:57.534 [rsSync] 31696787 objects cloned so far from collection newsblur.stories Thu Mar 21 00:34:13.257 [rsSync] clone newsblur.stories 31746431 Thu Mar 21 00:34:58.023 [rsSync] 31872682 objects cloned so far from collection newsblur.stories Thu Mar 21 00:35:13.167 [rsSync] clone newsblur.stories 31940479 Thu Mar 21 00:35:59.197 [rsSync] 32112585 objects cloned so far from collection newsblur.stories Thu Mar 21 00:36:11.025 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._39 Thu Mar 21 00:36:11.048 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._40 Thu Mar 21 00:36:13.084 [rsSync] clone newsblur.stories 32134911 Thu Mar 21 00:37:00.165 [rsSync] 32329755 objects cloned so far from collection newsblur.stories Thu Mar 21 00:37:13.819 [rsSync] clone newsblur.stories 32385663 Thu Mar 21 00:38:01.200 [rsSync] 32607631 objects cloned so far from collection newsblur.stories Thu Mar 21 00:38:13.041 [rsSync] clone newsblur.stories 32670463 Thu Mar 21 00:38:33.438 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.33, filling with zeroes... Thu Mar 21 00:38:33.445 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.33, size: 2047MB, took 0.007 secs Thu Mar 21 00:39:02.709 [rsSync] 32876109 objects cloned so far from collection newsblur.stories Thu Mar 21 00:39:13.003 [rsSync] clone newsblur.stories 32920063 Thu Mar 21 00:40:03.052 [rsSync] 33072959 objects cloned so far from collection newsblur.stories Thu Mar 21 00:40:13.109 [rsSync] clone newsblur.stories 33121151 Thu Mar 21 00:41:04.547 [rsSync] 33347025 objects cloned so far from collection newsblur.stories Thu Mar 21 00:41:13.008 [rsSync] clone newsblur.stories 33389311 Thu Mar 21 00:41:53.853 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._41 Thu Mar 21 00:42:05.138 [rsSync] 33541462 objects cloned so far from collection newsblur.stories Thu Mar 21 00:42:13.614 [rsSync] clone newsblur.stories 33560447 Thu Mar 21 00:43:06.061 [rsSync] 33750885 objects cloned so far from collection newsblur.stories Thu Mar 21 00:43:13.006 [rsSync] clone newsblur.stories 33779583 Thu Mar 21 00:43:29.585 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.34, filling with zeroes... Thu Mar 21 00:43:29.599 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.34, size: 2047MB, took 0.013 secs Thu Mar 21 00:44:08.236 [rsSync] 33925383 objects cloned so far from collection newsblur.stories Thu Mar 21 00:44:13.431 [rsSync] clone newsblur.stories 33941631 Thu Mar 21 00:45:09.005 [rsSync] 34087689 objects cloned so far from collection newsblur.stories Thu Mar 21 00:45:14.269 [rsSync] clone newsblur.stories 34094335 Thu Mar 21 00:45:33.305 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._42 Thu Mar 21 00:46:10.725 [rsSync] 34242756 objects cloned so far from collection newsblur.stories Thu Mar 21 00:46:14.528 [rsSync] clone newsblur.stories 34252799 Thu Mar 21 00:47:11.008 [rsSync] 34436304 objects cloned so far from collection newsblur.stories Thu Mar 21 00:47:14.153 [rsSync] clone newsblur.stories 34452479 Thu Mar 21 00:48:12.380 [rsSync] 34709989 objects cloned so far from collection newsblur.stories Thu Mar 21 00:48:14.357 [rsSync] clone newsblur.stories 34724095 Thu Mar 21 00:48:31.312 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._43 Thu Mar 21 00:48:35.038 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.35, filling with zeroes... Thu Mar 21 00:48:35.045 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.35, size: 2047MB, took 0.006 secs Thu Mar 21 00:49:13.584 [rsSync] 34983801 objects cloned so far from collection newsblur.stories Thu Mar 21 00:49:14.510 [rsSync] clone newsblur.stories 34985983 Thu Mar 21 00:50:15.174 [rsSync] clone newsblur.stories 35140095 Thu Mar 21 00:50:15.175 [rsSync] 35140138 objects cloned so far from collection newsblur.stories Thu Mar 21 00:51:15.471 [rsSync] clone newsblur.stories 35384447 Thu Mar 21 00:51:16.004 [rsSync] 35387661 objects cloned so far from collection newsblur.stories Thu Mar 21 00:51:24.654 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._44 Thu Mar 21 00:52:15.024 [rsSync] clone newsblur.stories 35577599 Thu Mar 21 00:52:17.579 [rsSync] 35590226 objects cloned so far from collection newsblur.stories Thu Mar 21 00:53:15.243 [rsSync] clone newsblur.stories 35860735 Thu Mar 21 00:53:18.178 [rsSync] 35872166 objects cloned so far from collection newsblur.stories Thu Mar 21 00:53:18.585 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.36, filling with zeroes... Thu Mar 21 00:53:18.592 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.36, size: 2047MB, took 0.006 secs Thu Mar 21 00:54:12.947 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._45 Thu Mar 21 00:54:12.969 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._46 Thu Mar 21 00:54:15.136 [rsSync] clone newsblur.stories 36104447 Thu Mar 21 00:54:19.261 [rsSync] 36129087 objects cloned so far from collection newsblur.stories Thu Mar 21 00:55:15.255 [rsSync] clone newsblur.stories 36377215 Thu Mar 21 00:55:20.579 [rsSync] 36399978 objects cloned so far from collection newsblur.stories Thu Mar 21 00:55:38.698 [rsSync] build index newsblur.stories { _id: 1 } Thu Mar 21 00:55:48.005 [rsSync] Index: (1/3) External Sort Progress: 1055200/36461889 2% Thu Mar 21 00:55:58.004 [rsSync] Index: (1/3) External Sort Progress: 2420700/36461889 6% Thu Mar 21 00:56:08.004 [rsSync] Index: (1/3) External Sort Progress: 3568000/36461889 9% Thu Mar 21 00:56:18.004 [rsSync] Index: (1/3) External Sort Progress: 4857300/36461889 13% Thu Mar 21 00:56:28.004 [rsSync] Index: (1/3) External Sort Progress: 5964700/36461889 16% Thu Mar 21 00:56:38.005 [rsSync] Index: (1/3) External Sort Progress: 7081900/36461889 19% Thu Mar 21 00:56:48.004 [rsSync] Index: (1/3) External Sort Progress: 8390300/36461889 23% Thu Mar 21 00:56:58.004 [rsSync] Index: (1/3) External Sort Progress: 9615100/36461889 26% Thu Mar 21 00:57:08.004 [rsSync] Index: (1/3) External Sort Progress: 10901400/36461889 29% Thu Mar 21 00:57:18.463 [rsSync] Index: (1/3) External Sort Progress: 12000000/36461889 32% Thu Mar 21 00:57:28.004 [rsSync] Index: (1/3) External Sort Progress: 13163500/36461889 36% Thu Mar 21 00:57:38.005 [rsSync] Index: (1/3) External Sort Progress: 14507900/36461889 39% Thu Mar 21 00:57:48.004 [rsSync] Index: (1/3) External Sort Progress: 15545200/36461889 42% Thu Mar 21 00:57:58.004 [rsSync] Index: (1/3) External Sort Progress: 16384400/36461889 44% Thu Mar 21 00:58:08.004 [rsSync] Index: (1/3) External Sort Progress: 17540100/36461889 48% Thu Mar 21 00:58:18.004 [rsSync] Index: (1/3) External Sort Progress: 18657500/36461889 51% Thu Mar 21 00:58:28.005 [rsSync] Index: (1/3) External Sort Progress: 19551600/36461889 53% Thu Mar 21 00:58:38.004 [rsSync] Index: (1/3) External Sort Progress: 20383800/36461889 55% Thu Mar 21 00:58:48.004 [rsSync] Index: (1/3) External Sort Progress: 21279000/36461889 58% Thu Mar 21 00:58:59.305 [rsSync] Index: (1/3) External Sort Progress: 22000000/36461889 60% Thu Mar 21 00:59:09.011 [rsSync] Index: (1/3) External Sort Progress: 22725800/36461889 62% Thu Mar 21 00:59:19.004 [rsSync] Index: (1/3) External Sort Progress: 23452700/36461889 64% Thu Mar 21 00:59:29.005 [rsSync] Index: (1/3) External Sort Progress: 24398300/36461889 66% Thu Mar 21 00:59:39.346 [rsSync] Index: (1/3) External Sort Progress: 25000000/36461889 68% Thu Mar 21 00:59:49.007 [rsSync] Index: (1/3) External Sort Progress: 25612000/36461889 70% Thu Mar 21 00:59:59.004 [rsSync] Index: (1/3) External Sort Progress: 26181100/36461889 71% Thu Mar 21 01:00:09.006 [rsSync] Index: (1/3) External Sort Progress: 26831500/36461889 73% Thu Mar 21 01:00:19.004 [rsSync] Index: (1/3) External Sort Progress: 27320600/36461889 74% Thu Mar 21 01:00:29.004 [rsSync] Index: (1/3) External Sort Progress: 27682500/36461889 75% Thu Mar 21 01:00:39.113 [rsSync] Index: (1/3) External Sort Progress: 28000000/36461889 76% Thu Mar 21 01:00:49.004 [rsSync] Index: (1/3) External Sort Progress: 28718000/36461889 78% Thu Mar 21 01:00:59.006 [rsSync] Index: (1/3) External Sort Progress: 29201800/36461889 80% Thu Mar 21 01:01:09.006 [rsSync] Index: (1/3) External Sort Progress: 29812600/36461889 81% Thu Mar 21 01:01:19.004 [rsSync] Index: (1/3) External Sort Progress: 30685400/36461889 84% Thu Mar 21 01:01:29.005 [rsSync] Index: (1/3) External Sort Progress: 31331300/36461889 85% Thu Mar 21 01:01:39.005 [rsSync] Index: (1/3) External Sort Progress: 32134600/36461889 88% Thu Mar 21 01:01:49.004 [rsSync] Index: (1/3) External Sort Progress: 33094700/36461889 90% Thu Mar 21 01:01:59.004 [rsSync] Index: (1/3) External Sort Progress: 33931300/36461889 93% Thu Mar 21 01:02:09.004 [rsSync] Index: (1/3) External Sort Progress: 34422200/36461889 94% Thu Mar 21 01:02:19.004 [rsSync] Index: (1/3) External Sort Progress: 35134100/36461889 96% Thu Mar 21 01:02:29.697 [rsSync] Index: (1/3) External Sort Progress: 36000000/36461889 98% Thu Mar 21 01:02:36.798 [rsSync] external sort used : 37 files in 418 secs Thu Mar 21 01:02:46.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 1867000/36461889 5% Thu Mar 21 01:02:56.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 3681500/36461889 10% Thu Mar 21 01:03:06.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 5813800/36461889 15% Thu Mar 21 01:03:16.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 7612900/36461889 20% Thu Mar 21 01:03:26.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 9700600/36461889 26% Thu Mar 21 01:03:36.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 11770600/36461889 32% Thu Mar 21 01:03:46.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 13716700/36461889 37% Thu Mar 21 01:03:56.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 15722900/36461889 43% Thu Mar 21 01:04:06.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 17671600/36461889 48% Thu Mar 21 01:04:16.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 19636600/36461889 53% Thu Mar 21 01:04:26.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 21702600/36461889 59% Thu Mar 21 01:04:36.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 23774600/36461889 65% Thu Mar 21 01:04:46.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 25663100/36461889 70% Thu Mar 21 01:04:56.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 27756000/36461889 76% Thu Mar 21 01:05:06.110 [rsSync] Index: (2/3) BTree Bottom Up Progress: 29761400/36461889 81% Thu Mar 21 01:05:16.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 31593100/36461889 86% Thu Mar 21 01:05:26.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 33722100/36461889 92% Thu Mar 21 01:05:36.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 35823000/36461889 98% Thu Mar 21 01:05:38.701 [rsSync] done building bottom layer, going to commit Thu Mar 21 01:05:50.042 [rsSync] fastBuildIndex dupsToDrop:837 Thu Mar 21 01:05:50.265 [rsSync] build index done. scanned 36461889 total records. 611.566 secs Thu Mar 21 01:05:50.269 [websvr] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 339678, after repl: 339678, at end: 339688 } Thu Mar 21 01:05:50.270 [conn45214] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 244465, after repl: 244465, at end: 244475 } Thu Mar 21 01:05:50.270 [conn45214] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:20 reslen:3446 248881ms Thu Mar 21 01:05:50.270 [conn45246] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 232487, after repl: 232487, at end: 232497 } Thu Mar 21 01:05:50.270 [conn45246] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:76 reslen:3446 236726ms Thu Mar 21 01:05:50.270 [conn45279] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 220451, after repl: 220451, at end: 220461 } Thu Mar 21 01:05:50.271 [conn45279] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 reslen:3446 224511ms Thu Mar 21 01:05:50.271 [conn45370] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 184655, after repl: 184655, at end: 184665 } Thu Mar 21 01:05:50.271 [conn45370] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 reslen:3446 188090ms Thu Mar 21 01:05:50.271 [conn45338] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 196571, after repl: 196571, at end: 196581 } Thu Mar 21 01:05:50.271 [websvr] Socket http response send() errno:32 Broken pipe 127.0.0.1:43084 Thu Mar 21 01:05:50.271 [conn45338] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33 reslen:3446 200261ms Thu Mar 21 01:05:50.274 [websvr] Socket http response send() errno:32 Broken pipe 127.0.0.1:43085 Thu Mar 21 01:05:53.284 [rsSync] build index newsblur.classifier_title { _id: 1 } Thu Mar 21 01:05:53.335 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:05:53.337 [rsSync] build index done. scanned 25008 total records. 0.052 secs Thu Mar 21 01:05:53.338 [websvr] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 2310, after repl: 2310, at end: 2310 } Thu Mar 21 01:05:54.860 [websvr] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 1170, after repl: 1170, at end: 1170 } Thu Mar 21 01:05:55.578 [conn45862] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:25 reslen:3106 356ms Thu Mar 21 01:05:56.468 [conn45863] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:34 reslen:3106 596ms Thu Mar 21 01:05:57.629 [conn45864] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:82 reslen:3106 869ms Thu Mar 21 01:06:01.009 [rsSync] build index newsblur.classifier_tag { _id: 1 } Thu Mar 21 01:06:01.486 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:06:01.496 [rsSync] build index done. scanned 209893 total records. 0.486 secs Thu Mar 21 01:06:02.173 [conn45883] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:81 reslen:3106 221ms Thu Mar 21 01:07:03.416 [rsSync] 235064 objects cloned so far from collection newsblur.userstories Thu Mar 21 01:07:15.839 [rsSync] clone newsblur.userstories 255871 Thu Mar 21 01:08:12.644 [rsSync] 638924 objects cloned so far from collection newsblur.userstories Thu Mar 21 01:08:15.590 [rsSync] clone newsblur.userstories 660607 Thu Mar 21 01:09:14.167 [rsSync] 1019289 objects cloned so far from collection newsblur.userstories Thu Mar 21 01:09:50.394 [rsSync] clone newsblur.userstories 1041023 Thu Mar 21 01:10:15.190 [rsSync] 1272527 objects cloned so far from collection newsblur.userstories Thu Mar 21 01:10:50.007 [rsSync] clone newsblur.userstories 1976703 Thu Mar 21 01:11:16.006 [rsSync] 2364981 objects cloned so far from collection newsblur.userstories Thu Mar 21 01:11:50.046 [rsSync] clone newsblur.userstories 3000063 Thu Mar 21 01:12:17.079 [rsSync] 3850754 objects cloned so far from collection newsblur.userstories Thu Mar 21 01:12:44.360 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._47 Thu Mar 21 01:12:50.005 [rsSync] clone newsblur.userstories 4308735 Thu Mar 21 01:13:18.005 [rsSync] 4976482 objects cloned so far from collection newsblur.userstories Thu Mar 21 01:13:31.188 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.37, filling with zeroes... Thu Mar 21 01:13:31.192 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.37, size: 2047MB, took 0.003 secs Thu Mar 21 01:13:50.006 [rsSync] clone newsblur.userstories 5586175 Thu Mar 21 01:14:21.745 [rsSync] 6000102 objects cloned so far from collection newsblur.userstories Thu Mar 21 01:14:50.007 [rsSync] clone newsblur.userstories 6743167 Thu Mar 21 01:15:16.774 [websvr] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 12127, after repl: 12127, at end: 12127 } Thu Mar 21 01:15:20.519 [conn47325] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:34 reslen:3106 469ms Thu Mar 21 01:15:22.832 [rsSync] 7137895 objects cloned so far from collection newsblur.userstories Thu Mar 21 01:15:50.418 [rsSync] clone newsblur.userstories 7723135 Thu Mar 21 01:16:23.007 [rsSync] 8152752 objects cloned so far from collection newsblur.userstories Thu Mar 21 01:16:50.025 [rsSync] clone newsblur.userstories 8756223 Thu Mar 21 01:17:27.737 [rsSync] 9434817 objects cloned so far from collection newsblur.userstories Thu Mar 21 01:17:44.311 [rsSync] build index newsblur.userstories { _id: 1 } Thu Mar 21 01:17:54.796 [rsSync] Index: (1/3) External Sort Progress: 5000000/9862394 50% Thu Mar 21 01:18:04.005 [rsSync] Index: (1/3) External Sort Progress: 9583800/9862394 97% Thu Mar 21 01:18:05.217 [rsSync] external sort used : 10 files in 20 secs Thu Mar 21 01:18:15.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 3602900/9862394 36% Thu Mar 21 01:18:25.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 7090500/9862394 71% Thu Mar 21 01:18:32.709 [rsSync] done building bottom layer, going to commit Thu Mar 21 01:18:35.343 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:18:35.394 [rsSync] build index done. scanned 9862394 total records. 51.082 secs Thu Mar 21 01:19:03.857 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._48 Thu Mar 21 01:19:36.007 [rsSync] 185071 objects cloned so far from collection newsblur.starred_stories Thu Mar 21 01:19:36.009 [rsSync] clone newsblur.starred_stories 185087 Thu Mar 21 01:19:51.062 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.38, filling with zeroes... Thu Mar 21 01:19:51.070 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.38, size: 2047MB, took 0.008 secs Thu Mar 21 01:20:20.546 [rsSync] build index newsblur.starred_stories { _id: 1 } Thu Mar 21 01:20:22.171 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:20:22.203 [rsSync] build index done. scanned 402677 total records. 1.657 secs Thu Mar 21 01:20:23.311 [rsSync] build index newsblur.classifier_author { _id: 1 } Thu Mar 21 01:20:23.356 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:20:23.357 [rsSync] build index done. scanned 22887 total records. 0.046 secs Thu Mar 21 01:20:25.289 [rsSync] build index newsblur.classifier_feed { _id: 1 } Thu Mar 21 01:20:25.372 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:20:25.374 [rsSync] build index done. scanned 40895 total records. 0.085 secs Thu Mar 21 01:20:25.375 [rsSync] build index newsblur.system.users { user: 1, userSource: 1 } Thu Mar 21 01:20:25.375 [rsSync] build index done. scanned 0 total records. 0 secs Thu Mar 21 01:20:25.442 [rsSync] build index newsblur.system.users { _id: 1 } Thu Mar 21 01:20:25.443 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:20:25.443 [rsSync] build index done. scanned 1 total records. 0.001 secs Thu Mar 21 01:20:25.486 [rsSync] build index newsblur.statistics { _id: 1 } Thu Mar 21 01:20:25.487 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:20:25.488 [rsSync] build index done. scanned 17 total records. 0.001 secs Thu Mar 21 01:21:25.260 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._49 Thu Mar 21 01:21:26.004 [rsSync] 551206 objects cloned so far from collection newsblur.feed_icons Thu Mar 21 01:21:26.006 [rsSync] clone newsblur.feed_icons 551295 Thu Mar 21 01:22:26.011 [rsSync] clone newsblur.feed_icons 996095 Thu Mar 21 01:22:29.037 [rsSync] 1000024 objects cloned so far from collection newsblur.feed_icons Thu Mar 21 01:22:35.642 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.39, filling with zeroes... Thu Mar 21 01:22:35.649 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.39, size: 2047MB, took 0.006 secs Thu Mar 21 01:23:26.216 [rsSync] clone newsblur.feed_icons 1199487 Thu Mar 21 01:23:30.095 [rsSync] 1211479 objects cloned so far from collection newsblur.feed_icons Thu Mar 21 01:23:43.877 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._50 Thu Mar 21 01:24:26.499 [rsSync] clone newsblur.feed_icons 1399679 Thu Mar 21 01:24:32.791 [rsSync] 1407059 objects cloned so far from collection newsblur.feed_icons Thu Mar 21 01:25:10.752 [rsSync] build index newsblur.feed_icons { _id: 1 } Thu Mar 21 01:25:19.610 [rsSync] fastBuildIndex dupsToDrop:2 Thu Mar 21 01:25:19.618 [rsSync] build index done. scanned 1537426 total records. 8.865 secs Thu Mar 21 01:25:19.886 [rsSync] build index newsblur.feedback { _id: 1 } Thu Mar 21 01:25:19.887 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:25:19.887 [rsSync] build index done. scanned 20 total records. 0.001 secs Thu Mar 21 01:25:19.907 [rsSync] build index newsblur.inline { _id: 1 } Thu Mar 21 01:25:19.907 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:25:19.908 [rsSync] build index done. scanned 16 total records. 0 secs Thu Mar 21 01:26:08.627 [rsSync] build index newsblur.feed_fetch_history { _id: 1 } Thu Mar 21 01:26:16.004 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:26:16.013 [rsSync] build index done. scanned 1634063 total records. 7.386 secs Thu Mar 21 01:26:44.209 [rsSync] build index newsblur.page_fetch_history { _id: 1 } Thu Mar 21 01:26:46.919 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:26:46.973 [rsSync] build index done. scanned 973985 total records. 2.763 secs Thu Mar 21 01:26:51.728 [rsSync] build index newsblur.social_profile { _id: 1 } Thu Mar 21 01:26:52.040 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:26:52.047 [rsSync] build index done. scanned 66955 total records. 0.318 secs Thu Mar 21 01:27:03.459 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.40, filling with zeroes... Thu Mar 21 01:27:03.464 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.40, size: 2047MB, took 0.005 secs Thu Mar 21 01:27:03.539 [rsSync] build index newsblur.shared_stories { _id: 1 } Thu Mar 21 01:27:03.807 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:27:03.812 [rsSync] build index done. scanned 45447 total records. 0.271 secs Thu Mar 21 01:27:07.262 [rsSync] build index newsblur.social_subscription { _id: 1 } Thu Mar 21 01:27:07.550 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:27:07.557 [rsSync] build index done. scanned 65250 total records. 0.294 secs Thu Mar 21 01:27:25.429 [rsSync] build index newsblur.social_services { _id: 1 } Thu Mar 21 01:27:25.573 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:27:25.577 [rsSync] build index done. scanned 60396 total records. 0.148 secs Thu Mar 21 01:27:25.626 [rsSync] build index newsblur.social_invites { _id: 1 } Thu Mar 21 01:27:25.628 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:27:25.628 [rsSync] build index done. scanned 734 total records. 0.002 secs Thu Mar 21 01:28:19.687 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._51 Thu Mar 21 01:28:19.703 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._52 Thu Mar 21 01:28:26.004 [rsSync] clone newsblur.feed_push_history 3660671 Thu Mar 21 01:28:26.005 [rsSync] 3660676 objects cloned so far from collection newsblur.feed_push_history Thu Mar 21 01:28:37.531 [rsSync] build index newsblur.feed_push_history { _id: 1 } Thu Mar 21 01:28:46.157 [rsSync] external sort used : 5 files in 8 secs Thu Mar 21 01:28:55.088 [rsSync] done building bottom layer, going to commit Thu Mar 21 01:28:55.754 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:28:55.806 [rsSync] build index done. scanned 4125852 total records. 18.274 secs Thu Mar 21 01:28:57.953 [rsSync] build index newsblur.interactions { _id: 1 } Thu Mar 21 01:28:58.076 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:28:58.079 [rsSync] build index done. scanned 28355 total records. 0.124 secs Thu Mar 21 01:29:10.772 [rsSync] build index newsblur.activities { _id: 1 } Thu Mar 21 01:29:11.783 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:29:11.800 [rsSync] build index done. scanned 355580 total records. 1.027 secs Thu Mar 21 01:29:12.565 [rsSync] build index newsblur.sent_emails { _id: 1 } Thu Mar 21 01:29:12.624 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:29:12.626 [rsSync] build index done. scanned 15182 total records. 0.059 secs Thu Mar 21 01:30:09.764 [rsSync] build index newsblur.uploaded_opml { _id: 1 } Thu Mar 21 01:30:10.078 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:30:10.081 [rsSync] build index done. scanned 18303 total records. 0.316 secs Thu Mar 21 01:30:10.347 [rsSync] build index newsblur.category { _id: 1 } Thu Mar 21 01:30:10.348 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:30:10.348 [rsSync] build index done. scanned 9 total records. 0.001 secs Thu Mar 21 01:30:10.439 [rsSync] build index newsblur.category_site { _id: 1 } Thu Mar 21 01:30:10.440 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:30:10.441 [rsSync] build index done. scanned 52 total records. 0.001 secs Thu Mar 21 01:30:10.561 [rsSync] build index newsblur.follow_request { _id: 1 } Thu Mar 21 01:30:10.563 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:30:10.563 [rsSync] build index done. scanned 53 total records. 0.001 secs Thu Mar 21 01:30:10.567 [rsSync] replSet initial sync cloning db: nbanalytics Thu Mar 21 01:30:10.596 [FileAllocator] allocating new datafile /var/lib/mongodb/nbanalytics.ns, filling with zeroes... Thu Mar 21 01:30:10.598 [FileAllocator] done allocating datafile /var/lib/mongodb/nbanalytics.ns, size: 16MB, took 0.002 secs Thu Mar 21 01:30:10.599 [FileAllocator] allocating new datafile /var/lib/mongodb/nbanalytics.0, filling with zeroes... Thu Mar 21 01:30:10.602 [FileAllocator] done allocating datafile /var/lib/mongodb/nbanalytics.0, size: 64MB, took 0.002 secs Thu Mar 21 01:30:10.603 [FileAllocator] allocating new datafile /var/lib/mongodb/nbanalytics.1, filling with zeroes... Thu Mar 21 01:30:10.605 [FileAllocator] done allocating datafile /var/lib/mongodb/nbanalytics.1, size: 128MB, took 0.002 secs Thu Mar 21 01:31:01.766 [FileAllocator] allocating new datafile /var/lib/mongodb/nbanalytics.2, filling with zeroes... Thu Mar 21 01:31:01.768 [FileAllocator] done allocating datafile /var/lib/mongodb/nbanalytics.2, size: 256MB, took 0.001 secs Thu Mar 21 01:31:11.005 [rsSync] 654752 objects cloned so far from collection nbanalytics.feed_fetches Thu Mar 21 01:31:26.274 [rsSync] clone nbanalytics.feed_fetches 681599 Thu Mar 21 01:31:34.825 [FileAllocator] allocating new datafile /var/lib/mongodb/nbanalytics.3, filling with zeroes... Thu Mar 21 01:31:34.828 [FileAllocator] done allocating datafile /var/lib/mongodb/nbanalytics.3, size: 512MB, took 0.002 secs Thu Mar 21 01:31:54.345 [rsSync] build index nbanalytics.feed_fetches { _id: 1 } Thu Mar 21 01:31:59.367 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:31:59.374 [rsSync] build index done. scanned 1132088 total records. 5.029 secs Thu Mar 21 01:32:30.734 [FileAllocator] allocating new datafile /var/lib/mongodb/nbanalytics.4, filling with zeroes... Thu Mar 21 01:32:30.737 [FileAllocator] done allocating datafile /var/lib/mongodb/nbanalytics.4, size: 1024MB, took 0.002 secs Thu Mar 21 01:32:33.801 [rsSync] build index nbanalytics.page_loads { _id: 1 } Thu Mar 21 01:32:36.009 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:32:36.049 [rsSync] build index done. scanned 803554 total records. 2.248 secs Thu Mar 21 01:32:36.050 [rsSync] replSet initial sync cloning db: test Thu Mar 21 01:32:36.071 [rsSync] replSet initial sync data copy, starting syncup Thu Mar 21 01:32:36.071 [rsSync] oplog sync 1 of 3 Thu Mar 21 01:32:37.038 [rsBackgroundSync] replSet syncing to: db05:27017 Thu Mar 21 01:32:38.756 [rsSyncNotifier] replset setting oplog notifier to db05:27017 Thu Mar 21 01:32:38.790 [repl writer worker 1] build index newsblur.classifier_title { user_id: 1, social_user_id: 1 } Thu Mar 21 01:32:38.958 [repl writer worker 1] build index done. scanned 25008 total records. 0.167 secs Thu Mar 21 01:32:38.973 [repl writer worker 1] build index newsblur.social_services { user_id: 1 } Thu Mar 21 01:32:39.311 [repl writer worker 1] build index done. scanned 60396 total records. 0.338 secs Thu Mar 21 01:32:39.358 [repl writer worker 1] build index newsblur.shared_stories { user_id: 1, shared_date: -1 } Thu Mar 21 01:32:39.820 [repl writer worker 1] build index done. scanned 45447 total records. 0.461 secs Thu Mar 21 01:32:39.845 [repl writer worker 1] build index newsblur.social_profile { following_user_ids: 1 } Thu Mar 21 01:32:40.410 [repl writer worker 1] build index done. scanned 66955 total records. 0.564 secs Thu Mar 21 01:32:40.483 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 01:32:40.491 [repl writer worker 1] build index newsblur.classifier_tag { social_user_id: 1 } Thu Mar 21 01:32:41.171 [repl writer worker 1] build index done. scanned 209893 total records. 0.679 secs Thu Mar 21 01:32:41.213 [repl writer worker 1] build index newsblur.feed_push_history { feed_id: 1 } Thu Mar 21 01:32:51.607 [repl writer worker 1] Index: (1/3) External Sort Progress: 3000000/4125852 72% Thu Mar 21 01:32:55.865 [repl writer worker 1] external sort used : 5 files in 14 secs Thu Mar 21 01:33:04.685 [repl writer worker 1] done building bottom layer, going to commit Thu Mar 21 01:33:05.361 [repl writer worker 1] build index done. scanned 4125852 total records. 24.147 secs Thu Mar 21 01:33:05.595 [repl writer worker 1] build index newsblur.classifier_author { user_id: 1, social_user_id: 1 } Thu Mar 21 01:33:05.672 [repl writer worker 1] build index done. scanned 22887 total records. 0.076 secs Thu Mar 21 01:33:05.674 [repl writer worker 1] build index newsblur.social_profile { username: 1 } Thu Mar 21 01:33:06.193 [repl writer worker 1] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:33:06.200 [repl writer worker 1] build index done. scanned 66955 total records. 0.525 secs Thu Mar 21 01:33:06.229 [repl writer worker 1] build index newsblur.feed_fetch_history { fetch_date: -1 } Thu Mar 21 01:33:13.397 [repl writer worker 1] build index done. scanned 1634063 total records. 7.167 secs Thu Mar 21 01:33:13.651 [repl writer worker 1] build index newsblur.follow_request { follower_user_id: 1, followee_user_id: 1 } Thu Mar 21 01:33:13.652 [repl writer worker 1] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:33:13.652 [repl writer worker 1] build index done. scanned 53 total records. 0.001 secs Thu Mar 21 01:33:13.658 [repl writer worker 1] build index newsblur.classifier_title { user_id: 1, feed_id: 1 } Thu Mar 21 01:33:13.787 [repl writer worker 1] build index done. scanned 25008 total records. 0.129 secs Thu Mar 21 01:33:13.806 [repl writer worker 1] build index newsblur.stories { story_feed_id: 1, story_date: -1 } Thu Mar 21 01:33:23.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 1420600/36461052 3% Thu Mar 21 01:33:33.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 2919800/36461052 8% Thu Mar 21 01:33:43.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 3965600/36461052 10% Thu Mar 21 01:33:53.152 [repl writer worker 1] Index: (1/3) External Sort Progress: 5000000/36461052 13% Thu Mar 21 01:34:03.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 6512800/36461052 17% Thu Mar 21 01:34:14.664 [repl writer worker 1] Index: (1/3) External Sort Progress: 8000000/36461052 21% Thu Mar 21 01:34:24.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 9065600/36461052 24% Thu Mar 21 01:34:34.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 10168900/36461052 27% Thu Mar 21 01:34:44.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 11119000/36461052 30% Thu Mar 21 01:34:54.010 [repl writer worker 1] Index: (1/3) External Sort Progress: 11743100/36461052 32% Thu Mar 21 01:35:04.012 [repl writer worker 1] Index: (1/3) External Sort Progress: 12262000/36461052 33% Thu Mar 21 01:35:14.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 12714200/36461052 34% Thu Mar 21 01:35:24.813 [repl writer worker 1] Index: (1/3) External Sort Progress: 13000000/36461052 35% Thu Mar 21 01:35:34.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 13591800/36461052 37% Thu Mar 21 01:35:44.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 14279800/36461052 39% Thu Mar 21 01:35:54.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 15152300/36461052 41% Thu Mar 21 01:36:04.832 [repl writer worker 1] Index: (1/3) External Sort Progress: 16000000/36461052 43% Thu Mar 21 01:36:14.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 16967500/36461052 46% Thu Mar 21 01:36:24.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 17810400/36461052 48% Thu Mar 21 01:36:34.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 18641500/36461052 51% Thu Mar 21 01:36:44.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 19547600/36461052 53% Thu Mar 21 01:36:54.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 20570800/36461052 56% Thu Mar 21 01:37:04.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 21340300/36461052 58% Thu Mar 21 01:37:14.347 [repl writer worker 1] Index: (1/3) External Sort Progress: 22000000/36461052 60% Thu Mar 21 01:37:24.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 22839500/36461052 62% Thu Mar 21 01:37:34.007 [repl writer worker 1] Index: (1/3) External Sort Progress: 23626100/36461052 64% Thu Mar 21 01:37:44.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 24081500/36461052 66% Thu Mar 21 01:37:54.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 24806000/36461052 68% Thu Mar 21 01:38:04.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 25182800/36461052 69% Thu Mar 21 01:38:14.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 25662000/36461052 70% Thu Mar 21 01:38:24.008 [repl writer worker 1] Index: (1/3) External Sort Progress: 26131100/36461052 71% Thu Mar 21 01:38:34.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 26595500/36461052 72% Thu Mar 21 01:38:44.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 27076100/36461052 74% Thu Mar 21 01:38:54.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 27519500/36461052 75% Thu Mar 21 01:39:04.234 [repl writer worker 1] Index: (1/3) External Sort Progress: 28000000/36461052 76% Thu Mar 21 01:39:14.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 28747000/36461052 78% Thu Mar 21 01:39:24.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 29261900/36461052 80% Thu Mar 21 01:39:35.501 [repl writer worker 1] Index: (1/3) External Sort Progress: 30000000/36461052 82% Thu Mar 21 01:39:45.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 30934500/36461052 84% Thu Mar 21 01:39:55.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 31574600/36461052 86% Thu Mar 21 01:40:05.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 32233400/36461052 88% Thu Mar 21 01:40:16.555 [repl writer worker 1] Index: (1/3) External Sort Progress: 33000000/36461052 90% Thu Mar 21 01:40:26.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 33688900/36461052 92% Thu Mar 21 01:40:36.006 [repl writer worker 1] Index: (1/3) External Sort Progress: 34066700/36461052 93% Thu Mar 21 01:40:46.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 34667300/36461052 95% Thu Mar 21 01:40:56.006 [repl writer worker 1] Index: (1/3) External Sort Progress: 35107500/36461052 96% Thu Mar 21 01:41:06.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 35824500/36461052 98% Thu Mar 21 01:41:16.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 36407800/36461052 99% Thu Mar 21 01:41:18.286 [repl writer worker 1] external sort used : 37 files in 484 secs Thu Mar 21 01:41:28.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 1994000/36461052 5% Thu Mar 21 01:41:38.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 4122200/36461052 11% Thu Mar 21 01:41:48.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 6313100/36461052 17% Thu Mar 21 01:41:56.316 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.41, filling with zeroes... Thu Mar 21 01:41:56.323 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.41, size: 2047MB, took 0.006 secs Thu Mar 21 01:41:58.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 8376800/36461052 22% Thu Mar 21 01:42:08.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 10333600/36461052 28% Thu Mar 21 01:42:18.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 12492300/36461052 34% Thu Mar 21 01:42:28.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 14598800/36461052 40% Thu Mar 21 01:42:38.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 16583300/36461052 45% Thu Mar 21 01:42:48.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 18579900/36461052 50% Thu Mar 21 01:42:58.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 20572600/36461052 56% Thu Mar 21 01:43:08.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 22528100/36461052 61% Thu Mar 21 01:43:10.382 [repl writer worker 1] DR101 latency warning on journal file open 640ms Thu Mar 21 01:43:18.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 24334600/36461052 66% Thu Mar 21 01:43:28.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 26404400/36461052 72% Thu Mar 21 01:43:38.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 28207000/36461052 77% Thu Mar 21 01:43:48.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 30170900/36461052 82% Thu Mar 21 01:43:58.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 32084000/36461052 87% Thu Mar 21 01:44:08.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 34144800/36461052 93% Thu Mar 21 01:44:18.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 36075700/36461052 98% Thu Mar 21 01:44:19.763 [repl writer worker 1] done building bottom layer, going to commit Thu Mar 21 01:44:31.253 [repl writer worker 1] build index done. scanned 36461052 total records. 677.446 secs Thu Mar 21 01:44:31.265 [conn50648] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 445756, after repl: 445756, at end: 445766 } Thu Mar 21 01:44:31.265 [conn50648] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:100 reslen:3802 453214ms Thu Mar 21 01:44:31.267 [conn50752] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 409827, after repl: 409827, at end: 409837 } Thu Mar 21 01:44:31.268 [conn50752] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 reslen:3802 416697ms Thu Mar 21 01:44:31.268 [conn50779] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 397818, after repl: 397818, at end: 397828 } Thu Mar 21 01:44:31.268 [websvr] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 552962, after repl: 552962, at end: 552972 } Thu Mar 21 01:44:31.269 [conn50779] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 reslen:3802 404506ms Thu Mar 21 01:44:31.269 [conn51533] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 107482, after repl: 107482, at end: 107492 } Thu Mar 21 01:44:31.269 [conn51533] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:84 reslen:3802 109502ms Thu Mar 21 01:44:31.269 [conn51379] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 167120, after repl: 167120, at end: 167130 } Thu Mar 21 01:44:31.269 [conn50688] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 433819, after repl: 433819, at end: 433829 } Thu Mar 21 01:44:31.269 [conn51379] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:52 reslen:3802 170313ms Thu Mar 21 01:44:31.269 [conn50688] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:910 reslen:3802 441100ms Thu Mar 21 01:44:31.269 [websvr] Socket http response send() errno:32 Broken pipe 127.0.0.1:43349 Thu Mar 21 01:44:31.269 [conn51440] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 143388, after repl: 143388, at end: 143398 } Thu Mar 21 01:44:31.270 [conn51440] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:365 reslen:3802 145999ms Thu Mar 21 01:44:31.270 [conn51502] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 119628, after repl: 119628, at end: 119638 } Thu Mar 21 01:44:31.270 [conn50616] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 457806, after repl: 457806, at end: 457816 } Thu Mar 21 01:44:31.270 [conn50616] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:1579 reslen:3802 465409ms Thu Mar 21 01:44:31.270 [conn51411] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 155188, after repl: 155188, at end: 155198 } Thu Mar 21 01:44:31.271 [conn51411] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:225 reslen:3802 158180ms Thu Mar 21 01:44:31.271 [conn51502] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:85 reslen:3802 121828ms Thu Mar 21 01:44:31.275 [conn50752] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:41197] Thu Mar 21 01:44:31.275 [conn50688] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:41194] Thu Mar 21 01:44:31.275 [conn50779] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:41200] Thu Mar 21 01:44:31.275 [conn50616] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:41191] Thu Mar 21 01:44:31.276 [conn50648] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:41192] Thu Mar 21 01:44:31.275 [conn50811] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:41201] Thu Mar 21 01:44:31.277 [websvr] Socket http response send() errno:32 Broken pipe 127.0.0.1:43351 Thu Mar 21 01:44:31.281 [websvr] Socket http response send() errno:32 Broken pipe 127.0.0.1:43355 Thu Mar 21 01:44:31.300 [conn50720] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:41196] Thu Mar 21 01:44:31.300 [conn50843] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:41203] Thu Mar 21 01:44:32.617 [repl writer worker 1] build index newsblur.classifier_title { social_user_id: 1 } Thu Mar 21 01:44:32.738 [repl writer worker 1] build index done. scanned 25008 total records. 0.12 secs Thu Mar 21 01:44:32.760 [repl writer worker 1] build index newsblur.userstories { user_id: 1, feed_id: 1, story_id: 1 } Thu Mar 21 01:44:42.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 1708900/9862394 17% Thu Mar 21 01:44:54.381 [repl writer worker 1] Index: (1/3) External Sort Progress: 2950100/9862394 29% Thu Mar 21 01:45:04.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 4099200/9862394 41% Thu Mar 21 01:45:14.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 5851000/9862394 59% Thu Mar 21 01:45:25.304 [repl writer worker 1] Index: (1/3) External Sort Progress: 6815300/9862394 69% Thu Mar 21 01:45:35.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 8614400/9862394 87% Thu Mar 21 01:45:45.715 [repl writer worker 1] Index: (1/3) External Sort Progress: 9685100/9862394 98% Thu Mar 21 01:45:47.222 [repl writer worker 1] external sort used : 11 files in 74 secs Thu Mar 21 01:45:57.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 2154100/9862394 21% Thu Mar 21 01:46:07.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 4229000/9862394 42% Thu Mar 21 01:46:17.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 5882200/9862394 59% Thu Mar 21 01:46:25.665 [repl writer worker 1] old journal file will be removed: /var/lib/mongodb/journal/j._53 Thu Mar 21 01:46:27.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 8033900/9862394 81% Thu Mar 21 01:46:34.350 [repl writer worker 1] done building bottom layer, going to commit Thu Mar 21 01:46:41.245 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.42, filling with zeroes... Thu Mar 21 01:46:41.250 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.42, size: 2047MB, took 0.005 secs Thu Mar 21 01:46:42.383 [repl writer worker 1] build index done. scanned 9862394 total records. 129.623 secs Thu Mar 21 01:46:42.385 [websvr] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 95307, after repl: 95307, at end: 95307 } Thu Mar 21 01:46:42.392 [conn52153] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:48 reslen:3462 886ms Thu Mar 21 01:46:43.300 [repl writer worker 1] build index newsblur.social_services { twitter_friend_ids: 1 } Thu Mar 21 01:46:50.689 [repl writer worker 1] external sort used : 32 files in 7 secs Thu Mar 21 01:47:00.879 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 1638500/1891481 86% Thu Mar 21 01:47:02.385 [repl writer worker 1] done building bottom layer, going to commit Thu Mar 21 01:47:02.469 [repl writer worker 1] build index done. scanned 60396 total records. 19.168 secs Thu Mar 21 01:47:02.471 [conn52200] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 6452, after repl: 6452, at end: 6452 } Thu Mar 21 01:47:02.471 [conn52200] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33 reslen:3802 6592ms Thu Mar 21 01:47:02.947 [repl writer worker 1] build index newsblur.social_services { facebook_friend_ids: 1 } Thu Mar 21 01:47:13.748 [repl writer worker 1] build index done. scanned 60396 total records. 10.801 secs Thu Mar 21 01:47:14.105 [repl writer worker 1] build index newsblur.feed_push_history { push_date: -1 } Thu Mar 21 01:47:23.199 [repl writer worker 1] external sort used : 5 files in 9 secs Thu Mar 21 01:47:31.568 [repl writer worker 1] done building bottom layer, going to commit Thu Mar 21 01:47:32.344 [repl writer worker 1] build index done. scanned 4125852 total records. 18.238 secs Thu Mar 21 01:47:32.651 [repl writer worker 1] build index newsblur.classifier_author { social_user_id: 1 } Thu Mar 21 01:47:32.720 [repl writer worker 1] build index done. scanned 22887 total records. 0.068 secs Thu Mar 21 01:47:32.745 [repl writer worker 1] build index newsblur.feed_fetch_history { fetch_date: 1, status_code: 1 } Thu Mar 21 01:47:40.982 [repl writer worker 1] build index done. scanned 1634063 total records. 8.236 secs Thu Mar 21 01:47:41.462 [repl writer worker 1] build index newsblur.follow_request { follower_user_id: 1 } Thu Mar 21 01:47:41.465 [repl writer worker 1] build index done. scanned 53 total records. 0.001 secs Thu Mar 21 01:47:41.489 [repl writer worker 1] build index newsblur.page_fetch_history { feed_id: 1, fetch_date: 1 } Thu Mar 21 01:47:46.551 [repl writer worker 1] build index done. scanned 973985 total records. 5.062 secs Thu Mar 21 01:47:46.659 [repl writer worker 1] build index newsblur.shared_stories { story_guid: 1 } Thu Mar 21 01:47:47.085 [repl writer worker 1] newsblur.system.indexes Btree::insert: key too large to index, skipping newsblur.shared_stories.$story_guid_1 1296 { : "http://news.yahoo.com/blogs/lookout/replyallcalypse-nyu-email-141306209.html?fb_action_ids=4975655872859%2C4129051465909%2C4964134862577%2C10152279975..." } Thu Mar 21 01:47:47.106 [repl writer worker 1] warning: not all entries were added to the index, probably some keys were too large Thu Mar 21 01:47:47.119 [repl writer worker 1] build index done. scanned 45447 total records. 0.459 secs Thu Mar 21 01:47:47.851 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 01:47:47.884 [repl writer worker 1] build index newsblur.shared_stories { user_id: 1, story_feed_id: 1 } Thu Mar 21 01:47:47.907 [rsBackgroundSync] repl: old cursor isDead, will initiate a new one Thu Mar 21 01:47:48.420 [repl writer worker 1] build index done. scanned 45447 total records. 0.535 secs Thu Mar 21 01:47:48.451 [repl writer worker 1] build index newsblur.social_services { twitter_uid: 1 } Thu Mar 21 01:47:48.793 [repl writer worker 1] build index done. scanned 60396 total records. 0.341 secs Thu Mar 21 01:47:48.810 [repl writer worker 1] build index newsblur.social_profile { follower_user_ids: 1 } Thu Mar 21 01:47:49.413 [repl writer worker 1] build index done. scanned 66955 total records. 0.602 secs Thu Mar 21 01:47:49.444 [repl writer worker 1] build index newsblur.classifier_title { feed_id: 1 } Thu Mar 21 01:47:49.535 [repl writer worker 1] build index done. scanned 25008 total records. 0.09 secs Thu Mar 21 01:47:49.586 [repl writer worker 1] build index newsblur.feed_fetch_history { feed_id: 1, status_code: 1 } Thu Mar 21 01:47:57.222 [repl writer worker 1] external sort used : 2 files in 7 secs Thu Mar 21 01:48:00.890 [repl writer worker 1] done building bottom layer, going to commit Thu Mar 21 01:48:00.967 [repl writer worker 1] build index done. scanned 1634063 total records. 11.38 secs Thu Mar 21 01:48:01.318 [repl writer worker 1] build index newsblur.userstories { feed_id: 1, story_id: 1 } Thu Mar 21 01:48:12.972 [repl writer worker 1] Index: (1/3) External Sort Progress: 2000000/9862394 20% Thu Mar 21 01:48:22.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 3998800/9862394 40% Thu Mar 21 01:48:32.369 [repl writer worker 1] Index: (1/3) External Sort Progress: 5000000/9862394 50% Thu Mar 21 01:48:45.928 [repl writer worker 1] Index: (1/3) External Sort Progress: 7000000/9862394 70% Thu Mar 21 01:48:55.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 8795700/9862394 89% Thu Mar 21 01:49:05.788 [repl writer worker 1] external sort used : 10 files in 64 secs Thu Mar 21 01:49:15.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 2447300/9862394 24% Thu Mar 21 01:49:25.076 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 5019300/9862394 50% Thu Mar 21 01:49:35.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 7708000/9862394 78% Thu Mar 21 01:49:43.915 [repl writer worker 1] done building bottom layer, going to commit Thu Mar 21 01:49:49.062 [repl writer worker 1] old journal file will be removed: /var/lib/mongodb/journal/j._54 Thu Mar 21 01:49:50.124 [repl writer worker 1] build index done. scanned 9862394 total records. 108.805 secs Thu Mar 21 01:49:50.559 [repl writer worker 1] build index newsblur.feedback { style: 1 } Thu Mar 21 01:49:50.562 [repl writer worker 1] build index done. scanned 20 total records. 0.003 secs Thu Mar 21 01:49:50.588 [repl writer worker 1] build index newsblur.social_services { facebook_uid: 1 } Thu Mar 21 01:49:50.778 [repl writer worker 1] build index done. scanned 60396 total records. 0.19 secs Thu Mar 21 01:49:50.800 [repl writer worker 1] build index newsblur.social_profile { unfollowed_user_ids: 1 } Thu Mar 21 01:49:51.209 [repl writer worker 1] build index done. scanned 66955 total records. 0.408 secs Thu Mar 21 01:49:51.289 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 01:49:51.299 [repl writer worker 1] build index newsblur.follow_request { followee_user_id: 1 } Thu Mar 21 01:49:51.301 [repl writer worker 1] build index done. scanned 53 total records. 0.001 secs Thu Mar 21 01:49:51.304 [repl writer worker 1] build index newsblur.userstories { feed_id: 1, story_date: 1 } Thu Mar 21 01:50:01.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 2303000/9862394 23% Thu Mar 21 01:50:11.030 [repl writer worker 1] Index: (1/3) External Sort Progress: 4000000/9862394 40% Thu Mar 21 01:50:22.229 [repl writer worker 1] Index: (1/3) External Sort Progress: 6000000/9862394 60% Thu Mar 21 01:50:32.611 [repl writer worker 1] Index: (1/3) External Sort Progress: 8000000/9862394 81% Thu Mar 21 01:50:42.548 [repl writer worker 1] external sort used : 10 files in 51 secs Thu Mar 21 01:50:52.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 3184500/9862394 32% Thu Mar 21 01:51:02.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 6606500/9862394 66% Thu Mar 21 01:51:11.755 [repl writer worker 1] done building bottom layer, going to commit Thu Mar 21 01:51:14.711 [repl writer worker 1] build index done. scanned 9862394 total records. 83.406 secs Thu Mar 21 01:51:14.711 [websvr] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 70487, after repl: 70487, at end: 70487 } Thu Mar 21 01:51:14.872 [repl writer worker 1] build index newsblur.classifier_tag { user_id: 1, feed_id: 1, social_user_id: 1, tag: 1 } Thu Mar 21 01:51:16.453 [repl writer worker 1] build index done. scanned 209893 total records. 1.58 secs Thu Mar 21 01:51:16.548 [repl writer worker 1] build index newsblur.shared_stories { story_feed_id: 1 } Thu Mar 21 01:51:16.765 [repl writer worker 1] build index done. scanned 45447 total records. 0.216 secs Thu Mar 21 01:51:16.791 [repl writer worker 1] build index newsblur.stories { story_hash: 1 } Thu Mar 21 01:51:26.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 1531300/36461052 4% Thu Mar 21 01:51:36.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 2991600/36461052 8% Thu Mar 21 01:51:46.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 4499700/36461052 12% Thu Mar 21 01:51:56.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 6491900/36461052 17% Thu Mar 21 01:52:06.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 8103300/36461052 22% Thu Mar 21 01:52:16.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 9719900/36461052 26% Thu Mar 21 01:52:26.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 10833300/36461052 29% Thu Mar 21 01:52:36.009 [repl writer worker 1] Index: (1/3) External Sort Progress: 11822500/36461052 32% Thu Mar 21 01:52:46.008 [repl writer worker 1] Index: (1/3) External Sort Progress: 12344300/36461052 33% Thu Mar 21 01:52:56.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 13821900/36461052 37% Thu Mar 21 01:53:06.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 14668800/36461052 40% Thu Mar 21 01:53:16.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 15982300/36461052 43% Thu Mar 21 01:53:26.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 17291800/36461052 47% Thu Mar 21 01:53:36.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 18694600/36461052 51% Thu Mar 21 01:53:46.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 20121200/36461052 55% Thu Mar 21 01:53:56.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 21503500/36461052 58% Thu Mar 21 01:54:06.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 22483600/36461052 61% Thu Mar 21 01:54:16.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 23175700/36461052 63% Thu Mar 21 01:54:26.017 [repl writer worker 1] Index: (1/3) External Sort Progress: 23824500/36461052 65% Thu Mar 21 01:54:36.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 24491000/36461052 67% Thu Mar 21 01:54:46.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 25361700/36461052 69% Thu Mar 21 01:54:56.006 [repl writer worker 1] Index: (1/3) External Sort Progress: 26103200/36461052 71% Thu Mar 21 01:55:06.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 26847200/36461052 73% Thu Mar 21 01:55:16.037 [repl writer worker 1] Index: (1/3) External Sort Progress: 27238500/36461052 74% Thu Mar 21 01:55:26.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 27789300/36461052 76% Thu Mar 21 01:55:36.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 28411400/36461052 77% Thu Mar 21 01:55:46.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 29136900/36461052 79% Thu Mar 21 01:55:56.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 29720900/36461052 81% Thu Mar 21 01:56:08.883 [repl writer worker 1] Index: (1/3) External Sort Progress: 30430600/36461052 83% Thu Mar 21 01:56:18.014 [repl writer worker 1] Index: (1/3) External Sort Progress: 31206800/36461052 85% Thu Mar 21 01:56:28.007 [repl writer worker 1] Index: (1/3) External Sort Progress: 31691200/36461052 86% Thu Mar 21 01:56:38.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 32330100/36461052 88% Thu Mar 21 01:56:48.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 33077800/36461052 90% Thu Mar 21 01:56:58.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 33596900/36461052 92% Thu Mar 21 01:57:08.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 34388300/36461052 94% Thu Mar 21 01:57:18.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 35077900/36461052 96% Thu Mar 21 01:57:28.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 35772600/36461052 98% Thu Mar 21 01:57:39.340 [repl writer worker 1] Index: (1/3) External Sort Progress: 36430600/36461052 99% Thu Mar 21 01:57:39.967 [repl writer worker 1] external sort used : 12 files in 383 secs Thu Mar 21 01:57:49.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 2669200/11030508 24% Thu Mar 21 01:57:54.877 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.43, filling with zeroes... Thu Mar 21 01:57:54.881 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.43, size: 2047MB, took 0.004 secs Thu Mar 21 01:57:59.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 5577300/11030508 50% Thu Mar 21 01:58:09.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 8645600/11030508 78% Thu Mar 21 01:58:17.897 [repl writer worker 1] done building bottom layer, going to commit Thu Mar 21 01:58:21.535 [repl writer worker 1] fastBuildIndex dupsToDrop:0 Thu Mar 21 01:58:21.606 [repl writer worker 1] build index done. scanned 36461052 total records. 424.814 secs Thu Mar 21 01:58:21.620 [conn52936] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 392917, after repl: 392917, at end: 392917 } Thu Mar 21 01:58:21.620 [conn52936] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:115 reslen:3802 400058ms Thu Mar 21 01:58:21.621 [conn53043] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 356840, after repl: 356840, at end: 356840 } Thu Mar 21 01:58:21.621 [conn52904] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 404664, after repl: 404664, at end: 404664 } Thu Mar 21 01:58:21.621 [conn53016] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 368693, after repl: 368693, at end: 368693 } Thu Mar 21 01:58:21.621 [conn52904] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:512 reslen:3802 412004ms Thu Mar 21 01:58:21.621 [conn53043] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:20 reslen:3802 363457ms Thu Mar 21 01:58:21.621 [conn53016] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:348 reslen:3802 375473ms Thu Mar 21 01:58:21.621 [conn52878] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 416807, after repl: 416807, at end: 416807 } Thu Mar 21 01:58:21.621 [conn52878] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:591 reslen:3802 424361ms Thu Mar 21 01:58:21.621 [websvr] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 194040, after repl: 194040, at end: 194040 } Thu Mar 21 01:58:21.621 [conn53706] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 98694, after repl: 98694, at end: 98694 } Thu Mar 21 01:58:21.621 [conn53706] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:26 reslen:3802 100539ms Thu Mar 21 01:58:21.622 [conn53877] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 38758, after repl: 38758, at end: 38758 } Thu Mar 21 01:58:21.622 [conn53877] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:61 reslen:3802 39741ms Thu Mar 21 01:58:21.622 [conn53770] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 74723, after repl: 74723, at end: 74733 } Thu Mar 21 01:58:21.622 [conn53770] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:71 reslen:3802 76246ms Thu Mar 21 01:58:21.622 [conn53738] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 86823, after repl: 86823, at end: 86833 } Thu Mar 21 01:58:21.622 [conn53738] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:73 reslen:3802 88513ms Thu Mar 21 01:58:21.622 [conn53837] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 50869, after repl: 50869, at end: 50879 } Thu Mar 21 01:58:21.622 [conn53837] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:216 reslen:3802 52064ms Thu Mar 21 01:58:21.626 [conn52904] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:41301] Thu Mar 21 01:58:21.626 [conn53016] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:41306] Thu Mar 21 01:58:21.626 [conn53074] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:41310] Thu Mar 21 01:58:21.626 [conn52878] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:41299] Thu Mar 21 01:58:21.627 [conn52936] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:41302] Thu Mar 21 01:58:21.627 [conn53043] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:41308] Thu Mar 21 01:58:21.643 [conn52968] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:41305] Thu Mar 21 01:58:21.644 [conn53106] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:41311] Thu Mar 21 01:58:22.429 [repl writer worker 1] build index nbanalytics.feed_fetches { date: 1 } Thu Mar 21 01:58:27.787 [repl writer worker 1] build index done. scanned 1132088 total records. 5.357 secs Thu Mar 21 01:58:28.194 [repl writer worker 1] build index nbanalytics.feed_fetches { feed_id: 1 } Thu Mar 21 01:58:35.035 [repl writer worker 1] build index done. scanned 1132088 total records. 6.841 secs Thu Mar 21 01:58:35.440 [repl writer worker 1] build index nbanalytics.feed_fetches { server: 1 } Thu Mar 21 01:58:42.241 [repl writer worker 1] build index done. scanned 1132088 total records. 6.8 secs Thu Mar 21 01:58:42.379 [repl writer worker 1] build index nbanalytics.feed_fetches { feed_code: 1 } Thu Mar 21 01:58:47.967 [repl writer worker 1] build index done. scanned 1132088 total records. 5.587 secs Thu Mar 21 01:58:48.038 [repl writer worker 1] build index newsblur.page_fetch_history { fetch_date: 1, status_code: 1 } Thu Mar 21 01:58:51.448 [repl writer worker 1] build index done. scanned 973985 total records. 3.409 secs Thu Mar 21 01:58:51.677 [repl writer worker 1] build index newsblur.social_profile { requested_follow_user_ids: 1 } Thu Mar 21 01:58:52.259 [repl writer worker 1] build index done. scanned 66955 total records. 0.582 secs Thu Mar 21 01:58:52.486 [repl writer worker 1] build index newsblur.userstories { feed_id: 1, read_date: -1 } Thu Mar 21 01:59:02.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 2017200/9862394 20% Thu Mar 21 01:59:12.647 [repl writer worker 1] Index: (1/3) External Sort Progress: 4000000/9862394 40% Thu Mar 21 01:59:23.494 [repl writer worker 1] Index: (1/3) External Sort Progress: 6000000/9862394 60% Thu Mar 21 01:59:34.398 [repl writer worker 1] Index: (1/3) External Sort Progress: 8000000/9862394 81% Thu Mar 21 01:59:44.747 [repl writer worker 1] external sort used : 10 files in 52 secs Thu Mar 21 01:59:54.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 3198500/9862394 32% Thu Mar 21 02:00:04.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 6641900/9862394 67% Thu Mar 21 02:00:13.517 [repl writer worker 1] done building bottom layer, going to commit Thu Mar 21 02:00:16.526 [repl writer worker 1] old journal file will be removed: /var/lib/mongodb/journal/j._55 Thu Mar 21 02:00:16.535 [repl writer worker 1] old journal file will be removed: /var/lib/mongodb/journal/j._56 Thu Mar 21 02:00:16.735 [repl writer worker 1] build index done. scanned 9862394 total records. 84.248 secs Thu Mar 21 02:00:16.739 [websvr] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 12288, after repl: 12288, at end: 12288 } Thu Mar 21 02:00:16.860 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:00:17.065 [repl writer worker 1] build index newsblur.page_fetch_history { feed_id: 1, status_code: 1 } Thu Mar 21 02:00:22.366 [repl writer worker 1] build index done. scanned 973985 total records. 5.3 secs Thu Mar 21 02:00:22.368 [websvr] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 5147, after repl: 5147, at end: 5147 } Thu Mar 21 02:00:22.456 [repl writer worker 1] build index newsblur.shared_stories { user_id: 1, story_db_id: 1 } Thu Mar 21 02:00:22.925 [repl writer worker 1] build index done. scanned 45447 total records. 0.468 secs Thu Mar 21 02:00:22.965 [repl writer worker 1] build index newsblur.classifier_tag { user_id: 1, feed_id: 1 } Thu Mar 21 02:00:23.969 [repl writer worker 1] build index done. scanned 209893 total records. 1.003 secs Thu Mar 21 02:00:24.123 [repl writer worker 1] build index newsblur.classifier_feed { user_id: 1, feed_id: 1, social_user_id: 1 } Thu Mar 21 02:00:24.350 [repl writer worker 1] build index done. scanned 40895 total records. 0.225 secs Thu Mar 21 02:00:24.567 [repl writer worker 1] build index newsblur.shared_stories { shared_date: 1 } Thu Mar 21 02:00:24.884 [repl writer worker 1] build index done. scanned 45447 total records. 0.316 secs Thu Mar 21 02:00:24.925 [repl writer worker 1] build index newsblur.classifier_tag { feed_id: 1 } Thu Mar 21 02:00:25.841 [repl writer worker 1] build index done. scanned 209893 total records. 0.916 secs Thu Mar 21 02:00:25.842 [conn54203] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:25 reslen:3462 864ms Thu Mar 21 02:00:26.071 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:00:26.071 [repl writer worker 1] build index nbanalytics.page_loads { path: 1 } Thu Mar 21 02:00:29.532 [repl writer worker 1] build index done. scanned 803554 total records. 3.46 secs Thu Mar 21 02:00:29.533 [conn54204] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 3161, after repl: 3161, at end: 3161 } Thu Mar 21 02:00:29.533 [conn54204] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:85 reslen:3802 3213ms Thu Mar 21 02:00:29.655 [repl writer worker 1] build index nbanalytics.page_loads { date: 1 } Thu Mar 21 02:00:31.763 [repl writer worker 1] build index done. scanned 803554 total records. 2.107 secs Thu Mar 21 02:00:31.764 [conn54215] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 1900, after repl: 1900, at end: 1900 } Thu Mar 21 02:00:31.764 [conn54215] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:30 reslen:3802 1918ms Thu Mar 21 02:00:31.940 [repl writer worker 1] build index nbanalytics.page_loads { platform: 1 } Thu Mar 21 02:00:34.988 [repl writer worker 1] build index done. scanned 803554 total records. 3.048 secs Thu Mar 21 02:00:35.253 [repl writer worker 1] build index nbanalytics.page_loads { user_id: 1 } Thu Mar 21 02:00:38.412 [repl writer worker 1] build index done. scanned 803554 total records. 3.158 secs Thu Mar 21 02:00:38.413 [conn54222] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 2621, after repl: 2621, at end: 2621 } Thu Mar 21 02:00:38.413 [conn54222] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33 reslen:3802 3056ms Thu Mar 21 02:00:38.578 [repl writer worker 1] build index nbanalytics.page_loads { server: 1 } Thu Mar 21 02:00:41.578 [repl writer worker 1] build index done. scanned 803554 total records. 2.999 secs Thu Mar 21 02:00:41.580 [conn54229] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 2632, after repl: 2632, at end: 2632 } Thu Mar 21 02:00:41.580 [conn54229] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:98 reslen:3802 2678ms Thu Mar 21 02:00:42.039 [repl writer worker 1] build index newsblur.classifier_feed { user_id: 1, feed_id: 1 } Thu Mar 21 02:00:42.288 [repl writer worker 1] build index done. scanned 40895 total records. 0.249 secs Thu Mar 21 02:00:42.433 [repl writer worker 1] build index newsblur.classifier_tag { user_id: 1, social_user_id: 1 } Thu Mar 21 02:00:43.820 [repl writer worker 1] build index done. scanned 209893 total records. 1.385 secs Thu Mar 21 02:00:43.903 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:00:43.922 [repl writer worker 1] build index newsblur.activities { user_id: 1, date: -1 } Thu Mar 21 02:00:45.831 [repl writer worker 1] build index done. scanned 355580 total records. 1.907 secs Thu Mar 21 02:00:46.018 [repl writer worker 1] build index newsblur.shared_stories { story_guid: 1, user_id: 1 } Thu Mar 21 02:00:46.633 [repl writer worker 1] newsblur.system.indexes Btree::insert: key too large to index, skipping newsblur.shared_stories.$story_guid_1_user_id_1 1302 { : "http://news.yahoo.com/blogs/lookout/replyallcalypse-nyu-email-141306209.html?fb_action_ids=4975655872859%2C4129051465909%2C4964134862577%2C10152279975...", : 17948 } Thu Mar 21 02:00:46.677 [repl writer worker 1] fastBuildIndex dupsToDrop:0 Thu Mar 21 02:00:46.697 [repl writer worker 1] build index done. scanned 45447 total records. 0.679 secs Thu Mar 21 02:00:46.751 [repl writer worker 1] build index newsblur.classifier_feed { feed_id: 1 } Thu Mar 21 02:00:46.929 [repl writer worker 1] build index done. scanned 40895 total records. 0.177 secs Thu Mar 21 02:00:47.222 [repl writer worker 1] build index newsblur.activities { category: 1 } Thu Mar 21 02:00:48.795 [repl writer worker 1] build index done. scanned 355580 total records. 1.572 secs Thu Mar 21 02:00:48.849 [repl writer worker 1] build index newsblur.classifier_feed { user_id: 1, social_user_id: 1 } Thu Mar 21 02:00:48.979 [repl writer worker 1] build index done. scanned 40895 total records. 0.13 secs Thu Mar 21 02:00:49.043 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:00:49.124 [repl writer worker 1] build index newsblur.interactions { user_id: 1, date: -1 } Thu Mar 21 02:00:49.238 [repl writer worker 1] build index done. scanned 28355 total records. 0.114 secs Thu Mar 21 02:00:49.299 [repl writer worker 1] build index newsblur.starred_stories { user_id: 1, story_guid: 1 } Thu Mar 21 02:00:59.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 254000/402677 63% Thu Mar 21 02:01:04.932 [repl writer worker 1] external sort used : 0 files in 15 secs Thu Mar 21 02:01:05.471 [repl writer worker 1] done building bottom layer, going to commit Thu Mar 21 02:01:06.094 [repl writer worker 1] fastBuildIndex dupsToDrop:0 Thu Mar 21 02:01:06.165 [repl writer worker 1] build index done. scanned 402677 total records. 16.865 secs Thu Mar 21 02:01:06.334 [rsSync] replSet initialSyncOplogApplication applied 1001 operations, synced to Mar 20 20:07:36:40 Thu Mar 21 02:01:06.341 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:06.409 [repl writer worker 1] build index newsblur.interactions { category: 1 } Thu Mar 21 02:01:06.552 [repl writer worker 1] build index done. scanned 28355 total records. 0.142 secs Thu Mar 21 02:01:06.617 [repl writer worker 1] build index newsblur.starred_stories { user_id: 1, starred_date: -1 } Thu Mar 21 02:01:08.757 [repl writer worker 1] build index done. scanned 402677 total records. 2.139 secs Thu Mar 21 02:01:09.090 [repl writer worker 1] build index newsblur.activities { with_user_id: 1 } Thu Mar 21 02:01:10.386 [repl writer worker 1] build index done. scanned 355580 total records. 1.295 secs Thu Mar 21 02:01:10.791 [repl writer worker 1] build index newsblur.starred_stories { user_id: 1, story_feed_id: 1 } Thu Mar 21 02:01:13.209 [repl writer worker 1] build index done. scanned 402677 total records. 2.417 secs Thu Mar 21 02:01:13.452 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:13.457 [repl writer worker 1] build index newsblur.classifier_feed { social_user_id: 1 } Thu Mar 21 02:01:13.611 [repl writer worker 1] build index done. scanned 40895 total records. 0.154 secs Thu Mar 21 02:01:13.765 [repl writer worker 1] build index newsblur.statistics { key: 1 } Thu Mar 21 02:01:13.767 [repl writer worker 1] build index done. scanned 17 total records. 0.001 secs Thu Mar 21 02:01:14.033 [repl writer worker 1] build index newsblur.interactions { with_user_id: 1 } Thu Mar 21 02:01:14.192 [repl writer worker 1] build index done. scanned 28355 total records. 0.158 secs Thu Mar 21 02:01:14.208 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:14.283 [repl writer worker 1] build index newsblur.classifier_author { user_id: 1, feed_id: 1, social_user_id: 1, author: 1 } Thu Mar 21 02:01:14.522 [repl writer worker 1] build index done. scanned 22887 total records. 0.237 secs Thu Mar 21 02:01:14.810 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:15.125 [repl writer worker 1] build index newsblur.starred_stories { story_feed_id: 1 } Thu Mar 21 02:01:17.095 [repl writer worker 1] build index done. scanned 402677 total records. 1.97 secs Thu Mar 21 02:01:17.240 [repl writer worker 1] build index newsblur.classifier_author { user_id: 1, feed_id: 1 } Thu Mar 21 02:01:17.322 [repl writer worker 1] build index done. scanned 22887 total records. 0.081 secs Thu Mar 21 02:01:17.394 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:17.499 [repl writer worker 1] build index newsblur.classifier_author { feed_id: 1 } Thu Mar 21 02:01:17.563 [repl writer worker 1] build index done. scanned 22887 total records. 0.064 secs Thu Mar 21 02:01:17.704 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:17.734 [repl writer worker 1] build index newsblur.social_subscription { subscription_user_id: 1, user_id: 1 } Thu Mar 21 02:01:18.272 [repl writer worker 1] build index done. scanned 65250 total records. 0.538 secs Thu Mar 21 02:01:18.605 [repl writer worker 1] build index newsblur.social_profile { user_id: 1 } Thu Mar 21 02:01:19.010 [repl writer worker 1] fastBuildIndex dupsToDrop:0 Thu Mar 21 02:01:19.018 [repl writer worker 1] build index done. scanned 66955 total records. 0.412 secs Thu Mar 21 02:01:19.174 [repl writer worker 1] build index newsblur.social_subscription { user_id: 1, subscription_user_id: 1 } Thu Mar 21 02:01:19.678 [repl writer worker 1] build index done. scanned 65250 total records. 0.503 secs Thu Mar 21 02:01:19.780 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:20.167 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:20.409 [rsSync] replSet initialSyncOplogApplication applied 2002 operations, synced to Mar 20 20:07:49:32 Thu Mar 21 02:01:20.471 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:20.789 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:21.182 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:21.561 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:21.858 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:21.925 [repl writer worker 1] build index newsblur.category { title: 1 } Thu Mar 21 02:01:21.926 [repl writer worker 1] build index done. scanned 9 total records. 0.001 secs Thu Mar 21 02:01:22.040 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:22.371 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:22.659 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:22.970 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:23.247 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:23.583 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:23.818 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:24.111 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:24.290 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:24.496 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:24.810 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:25.008 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:25.194 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 02:01:25.229 [repl writer worker 1] ERROR: exception: E11000 duplicate key error index: newsblur.stories.$story_hash_1 dup key: { : "980568:65f04f" } on: { ts: Timestamp 1363810091000|32, h: -7663483269809030053, op: "i", ns: "newsblur.stories", o: { _id: ObjectId('514a172a0f55a7137913adea'), share_user_ids: {}, story_author_name: "", story_date: new Date(1363809963000), story_tags: {}, story_hash: "980568:65f04f", story_feed_id: 980568, story_permalink: "http://feedproxy.google.com/~r/DrudgeReportFeed/~3/Kn0_WSFPhAI/us-syria-crisis-lebanon-idUSBRE92J0AY20130320", story_title: "More Syrian shells land in Lebanon...", comment_user_ids: {}, story_content_z: BinData, story_guid: "http://www.reuters.com/article/2013/03/20/us-syria-crisis-lebanon-idUSBRE92J0AY20130320" } } Thu Mar 21 02:01:25.230 [repl writer worker 1] Fatal Assertion 16361 0xdcae01 0xd8ab83 0xc1ecb6 0xd986b1 0xe13709 0x7f1fac48fe9a 0x7f1fab7a2cbd /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdcae01] /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xd8ab83] /usr/bin/mongod(_ZN5mongo7replset21multiInitialSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x166) [0xc1ecb6] /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xd986b1] /usr/bin/mongod() [0xe13709] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f1fac48fe9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f1fab7a2cbd] Thu Mar 21 02:01:25.238 [repl writer worker 1] ***aborting after fassert() failure Thu Mar 21 02:01:25.238 Got signal: 6 (Aborted). Thu Mar 21 02:01:25.242 Backtrace: 0xdcae01 0x6ce879 0x7f1fab6e54a0 0x7f1fab6e5425 0x7f1fab6e8b8b 0xd8abbe 0xc1ecb6 0xd986b1 0xe13709 0x7f1fac48fe9a 0x7f1fab7a2cbd /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdcae01] /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6ce879] /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f1fab6e54a0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f1fab6e5425] /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f1fab6e8b8b] /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xd8abbe] /usr/bin/mongod(_ZN5mongo7replset21multiInitialSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x166) [0xc1ecb6] /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xd986b1] /usr/bin/mongod() [0xe13709] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f1fac48fe9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f1fab7a2cbd] ***** SERVER RESTARTED ***** Thu Mar 21 02:02:33.225 [initandlisten] MongoDB starting : pid=30796 port=27017 dbpath=/var/lib/mongodb 64-bit host=db20 Thu Mar 21 02:02:33.227 [initandlisten] db version v2.4.0 Thu Mar 21 02:02:33.227 [initandlisten] git version: ce2d666c04b4a80af58e8bbb3388b0680e8cfeb6 Thu Mar 21 02:02:33.227 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49 Thu Mar 21 02:02:33.227 [initandlisten] allocator: tcmalloc Thu Mar 21 02:02:33.227 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log", noauth: "true", replSet: "nbset", rest: "true", slowms: 100 } Thu Mar 21 02:02:33.443 [initandlisten] journal dir=/var/lib/mongodb/journal Thu Mar 21 02:02:33.443 [initandlisten] recover begin Thu Mar 21 02:02:33.443 [initandlisten] recover lsn: 20824747 Thu Mar 21 02:02:33.443 [initandlisten] recover /var/lib/mongodb/journal/j._57 Thu Mar 21 02:02:33.445 [initandlisten] recover skipping application of section seq:20766214 < lsn:20824747 Thu Mar 21 02:02:34.176 [initandlisten] recover cleaning up Thu Mar 21 02:02:34.176 [initandlisten] removeJournalFiles Thu Mar 21 02:02:34.187 [initandlisten] recover done Thu Mar 21 02:02:34.310 [websvr] admin web console waiting for connections on port 28017 Thu Mar 21 02:02:34.310 [initandlisten] waiting for connections on port 27017 Thu Mar 21 02:02:34.454 [rsStart] replSet I am db20:27017 Thu Mar 21 02:02:34.454 [rsStart] replSet STARTUP2 Thu Mar 21 02:02:34.458 [rsSync] replSet initial sync pending Thu Mar 21 02:02:34.458 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync Thu Mar 21 02:02:34.478 [rsHealthPoll] replSet member db05:27017 is up Thu Mar 21 02:02:34.478 [rsHealthPoll] replSet member db05:27017 is now in state SECONDARY Thu Mar 21 02:02:34.479 [rsHealthPoll] replSet member db04:27017 is up Thu Mar 21 02:02:34.479 [rsHealthPoll] replSet member db04:27017 is now in state PRIMARY Thu Mar 21 02:02:39.461 [rsBackgroundSync] replSet syncing to: db04:27017 Thu Mar 21 02:02:48.976 got signal 15 (Terminated), will terminate after current cmd ends Thu Mar 21 02:03:09.484 [rsBackgroundSync] Socket recv() timeout 199.15.249.98:27017 Thu Mar 21 02:03:09.484 [rsBackgroundSync] SocketException: remote: 199.15.249.98:27017 error: 9001 socket exception [3] server [199.15.249.98:27017] Thu Mar 21 02:03:09.484 [rsBackgroundSync] DBClientCursor::init call() failed Thu Mar 21 02:03:09.484 [rsSyncNotifier] replset setting oplog notifier to db04:27017 Thu Mar 21 02:03:09.508 [rsSync] replSet initial sync pending Thu Mar 21 02:03:09.508 [interruptThread] now exiting Thu Mar 21 02:03:09.508 Thu Mar 21 02:03:09.508 [rsBackgroundSync] replSet syncing to: db04:27017 dbexit: Thu Mar 21 02:03:09.508 [interruptThread] shutdown: going to close listening sockets... Thu Mar 21 02:03:09.508 [interruptThread] closing listening socket: 17 Thu Mar 21 02:03:09.508 [interruptThread] closing listening socket: 18 Thu Mar 21 02:03:09.508 [interruptThread] closing listening socket: 19 Thu Mar 21 02:03:09.508 [interruptThread] removing socket file: /tmp/mongodb-27017.sock Thu Mar 21 02:03:09.508 [interruptThread] shutdown: going to flush diaglog... Thu Mar 21 02:03:09.509 [interruptThread] shutdown: going to close sockets... Thu Mar 21 02:03:09.509 [interruptThread] shutdown: waiting for fs preallocator... Thu Mar 21 02:03:09.509 [interruptThread] shutdown: lock for final commit... Thu Mar 21 02:03:09.509 [rsBackgroundSync] repl: couldn't connect to server db04:27017 Thu Mar 21 02:03:09.510 [interruptThread] shutdown: final commit... Thu Mar 21 02:03:09.510 [rsBackgroundSync] replSet syncing to: db05:27017 Thu Mar 21 02:03:09.510 [interruptThread] shutdown: closing all files... Thu Mar 21 02:03:09.512 [interruptThread] closeAllFiles() finished Thu Mar 21 02:03:09.512 [interruptThread] journalCleanup... Thu Mar 21 02:03:09.512 [interruptThread] removeJournalFiles Thu Mar 21 02:03:09.515 [interruptThread] shutdown: removing fs lock... Thu Mar 21 02:03:09.515 dbexit: really exiting now