Mon Feb 25 03:30:15 [conn25583] end connection 172.17.16.22:19352 (191 connections now open) Mon Feb 25 03:30:15 [initandlisten] connection accepted from 172.17.16.22:19354 #25591 (192 connections now open) Mon Feb 25 03:30:15 [conn25591] authenticate db: local { authenticate: 1, nonce: "6b080e6616eb23f7", user: "__system", key: "49ed7c27e55ffc6503410d89904a91c5" } Mon Feb 25 03:30:15 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:64 reslen:2645 103ms Mon Feb 25 03:30:16 [conn25590] getmore mlog.ordertrack_worker-1201212 query: { query: {}, $snapshot: true } cursorid:4838168199749890290 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 4 locks(micros) r:709796 nreturned:345 reslen:4194674 378ms Mon Feb 25 03:30:16 [conn25590] getmore mlog.ordertrack_worker-1201212 query: { query: {}, $snapshot: true } cursorid:4838168199749890290 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:308053 nreturned:347 reslen:4200531 308ms Mon Feb 25 03:30:16 [conn191] command mlog.$cmd command: { count: "cartweb201302", query: { info.0: { $regex: /^.*timeout.*$/i }, now: { $gte: new Date(1361733900003), $lte: new Date(1361734200003) }, $or: [ { level: 40000 } ] } } ntoreturn:1 keyUpdates:0 numYields: 104 locks(micros) r:6055875 reslen:48 3275ms Mon Feb 25 03:30:17 [conn25590] getmore mlog.ordertrack_worker-1201212 query: { query: {}, $snapshot: true } cursorid:4838168199749890290 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 2 locks(micros) r:788704 nreturned:330 reslen:4196960 405ms Mon Feb 25 03:30:18 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:78 reslen:2645 274ms Mon Feb 25 03:30:18 [conn25590] getmore mlog.ordertrack_worker-1201212 query: { query: {}, $snapshot: true } cursorid:4838168199749890290 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 7 locks(micros) r:756140 nreturned:330 reslen:4196960 379ms Mon Feb 25 03:30:19 [conn25590] getmore mlog.ordertrack_worker-1201212 query: { query: {}, $snapshot: true } cursorid:4838168199749890290 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:435034 nreturned:331 reslen:4197161 435ms Mon Feb 25 03:30:19 [conn191] command mlog.$cmd command: { count: "cartweb201302", query: { info.0: { $regex: /^.*timeout.*$/i }, now: { $gte: new Date(1361733900003), $lte: new Date(1361734200003) }, $or: [ { level: 40000 } ] } } ntoreturn:1 keyUpdates:0 numYields: 110 locks(micros) r:4222707 reslen:48 2489ms Mon Feb 25 03:30:19 [conn25590] getmore mlog.ordertrack_worker-1201212 query: { query: {}, $snapshot: true } cursorid:4838168199749890290 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:382451 nreturned:348 reslen:4199443 382ms Mon Feb 25 03:30:20 [conn25590] getmore mlog.ordertrack_worker-1201212 query: { query: {}, $snapshot: true } cursorid:4838168199749890290 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:385177 nreturned:330 reslen:4196960 385ms Mon Feb 25 03:30:21 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:69 reslen:2645 109ms Mon Feb 25 03:30:21 [conn25590] getmore mlog.ordertrack_worker-1201212 query: { query: {}, $snapshot: true } cursorid:4838168199749890290 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 6 locks(micros) r:801777 nreturned:330 reslen:4196960 410ms Mon Feb 25 03:30:21 [conn191] query mlog.cartweb201302 query: { info.0: { $regex: /^.*timeout.*$/i }, now: { $gte: new Date(1361733900003), $lte: new Date(1361734200003) }, $or: [ { level: 40000 } ] } ntoreturn:500 ntoskip:0 nscanned:4553783 keyUpdates:0 numYields: 65 locks(micros) r:4470101 nreturned:0 reslen:20 2464ms Mon Feb 25 03:30:22 [conn25590] getmore mlog.ordertrack_worker-1201212 query: { query: {}, $snapshot: true } cursorid:4838168199749890290 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 6 locks(micros) r:775359 nreturned:330 reslen:4196960 395ms Mon Feb 25 03:30:22 [conn25590] getmore mlog.ordertrack_worker-1201212 query: { query: {}, $snapshot: true } cursorid:4838168199749890290 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:410156 nreturned:330 reslen:4196960 410ms Mon Feb 25 03:30:23 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:106 reslen:2625 207ms Mon Feb 25 03:30:23 [conn25590] getmore mlog.ordertrack_worker-1201212 query: { query: {}, $snapshot: true } cursorid:4838168199749890290 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 6 locks(micros) r:748980 nreturned:345 reslen:4205807 375ms Mon Feb 25 03:30:24 [conn25590] getmore mlog.ordertrack_worker-1201212 query: { query: {}, $snapshot: true } cursorid:4838168199749890290 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:720329 nreturned:345 reslen:4205807 392ms Mon Feb 25 03:30:25 [conn25590] getmore mlog.ordertrack_worker-1201212 query: { query: {}, $snapshot: true } cursorid:4838168199749890290 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:442072 nreturned:330 reslen:4196960 442ms Mon Feb 25 03:30:25 [conn25590] getmore mlog.ordertrack_worker-1201212 query: { query: {}, $snapshot: true } cursorid:4838168199749890290 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:376827 nreturned:330 reslen:4196960 376ms Mon Feb 25 03:30:26 [conn25590] getmore mlog.ordertrack_worker-1201212 query: { query: {}, $snapshot: true } cursorid:4838168199749890290 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:514036 nreturned:330 reslen:4196960 258ms Mon Feb 25 03:30:26 [conn25588] end connection 172.17.16.23:25327 (191 connections now open) Mon Feb 25 03:30:26 [initandlisten] connection accepted from 172.17.16.23:25334 #25592 (192 connections now open) Mon Feb 25 03:30:26 [conn25592] authenticate db: local { authenticate: 1, nonce: "cf152e6398ff4c40", user: "__system", key: "44d7e7deeb7ff00195e6069bc489452f" } Mon Feb 25 03:30:27 [conn25590] getmore mlog.ordertoolbar201212 query: { query: {}, $snapshot: true } cursorid:8352012374901740272 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 16 locks(micros) r:362070 nreturned:1555 reslen:4197861 410ms Mon Feb 25 03:30:29 [conn25590] getmore mlog.ordertoolbar201212 query: { query: {}, $snapshot: true } cursorid:8352012374901740272 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 25 locks(micros) r:165935 nreturned:1614 reslen:4198963 1227ms Mon Feb 25 03:30:30 [conn25590] getmore mlog.ordertoolbar201212 query: { query: {}, $snapshot: true } cursorid:8352012374901740272 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 16 locks(micros) r:882705 nreturned:1641 reslen:4196593 1552ms Mon Feb 25 03:30:45 [conn25591] end connection 172.17.16.22:19354 (191 connections now open) Mon Feb 25 03:30:45 [initandlisten] connection accepted from 172.17.16.22:19358 #25593 (192 connections now open) Mon Feb 25 03:30:45 [conn25593] authenticate db: local { authenticate: 1, nonce: "5595d6edca52832f", user: "__system", key: "8f675112a5b46ac49f6265d314330d1d" } Mon Feb 25 03:30:49 [initandlisten] connection accepted from 192.168.137.100:28846 #25594 (193 connections now open) Mon Feb 25 03:30:49 [conn25594] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "9da821c1af18b6f1", key: "a80edeae09d10cd3d7c90f8c7930213b" } Mon Feb 25 03:30:49 [conn25594] end connection 192.168.137.100:28846 (192 connections now open) Mon Feb 25 03:30:49 [initandlisten] connection accepted from 192.168.137.100:28847 #25595 (193 connections now open) Mon Feb 25 03:30:49 [conn25595] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "588f367f9da931e5", key: "9da716a4fe7a92a3fb680e6485e7a827" } Mon Feb 25 03:30:49 [conn25595] end connection 192.168.137.100:28847 (192 connections now open) Mon Feb 25 03:30:49 [initandlisten] connection accepted from 192.168.137.100:28848 #25596 (193 connections now open) Mon Feb 25 03:30:49 [conn25596] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "93e95643346cd1c5", key: "25965e2e49b680665acdd8df597eee24" } Mon Feb 25 03:30:49 [conn25596] end connection 192.168.137.100:28848 (192 connections now open) Mon Feb 25 03:30:49 [initandlisten] connection accepted from 192.168.137.100:28849 #25597 (193 connections now open) Mon Feb 25 03:30:49 [conn25597] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "b311540cc383253d", key: "763810892c6fd6642c281e6fe910d51c" } Mon Feb 25 03:30:49 [conn25597] end connection 192.168.137.100:28849 (192 connections now open) Mon Feb 25 03:30:50 [rsSync] info DFM::findAll(): extent 30:76aad000 was empty, skipping ahead. ns:local.replset.minvalid Mon Feb 25 03:30:50 [conn25590] getmore mlog.ordertrack_worker201302 query: { query: {}, $snapshot: true } cursorid:6869476577280760593 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 13 locks(micros) r:333034 nreturned:330 reslen:4196960 401ms Mon Feb 25 03:30:50 [conn25590] getmore mlog.ordertrack_worker201302 query: { query: {}, $snapshot: true } cursorid:6869476577280760593 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 15 locks(micros) r:107622 nreturned:353 reslen:4194932 239ms Mon Feb 25 03:30:55 [conn25391] serverStatus was very slow: { after basic: 0, middle of mem: 3430, after mem: 3430, after connections: 3430, after extra info: 3430, after counters: 3430, after repl: 3430, after asserts: 3430, after dur: 3440, at end: 3440 } Mon Feb 25 03:30:55 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:25 reslen:2816 3459ms Mon Feb 25 03:30:55 [conn25590] getmore mlog.ordertrack_worker201302 query: { query: {}, $snapshot: true } cursorid:6869476577280760593 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 12 locks(micros) r:988055 nreturned:330 reslen:4196960 4393ms Mon Feb 25 03:30:56 [conn25592] end connection 172.17.16.23:25334 (191 connections now open) Mon Feb 25 03:30:56 [initandlisten] connection accepted from 172.17.16.23:25338 #25598 (192 connections now open) Mon Feb 25 03:30:56 [conn25598] authenticate db: local { authenticate: 1, nonce: "8e5e18b1f525d268", user: "__system", key: "79ea39189fec1ad3e57c036529e6bda6" } Mon Feb 25 03:31:01 [conn25590] getmore mlog.ordertrack_worker201302 query: { query: {}, $snapshot: true } cursorid:6869476577280760593 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 2 locks(micros) r:295208 nreturned:330 reslen:4196960 198ms Mon Feb 25 03:31:01 [initandlisten] connection accepted from 127.0.0.1:41128 #25599 (193 connections now open) Mon Feb 25 03:31:01 [conn25599] authenticate db: admin { authenticate: 1, nonce: "80ac94f0272ca235", user: "admin", key: "060676fa73a5d5422512b0b1a40b56fe" } Mon Feb 25 03:31:01 [conn25599] authenticate db: admin { authenticate: 1, user: "admin", nonce: "699e3f74c8bee9fe", key: "fc5cafcad37284e0a61b6715bfcb1423" } Mon Feb 25 03:31:01 [conn25599] end connection 127.0.0.1:41128 (192 connections now open) Mon Feb 25 03:31:02 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:478120 nreturned:9355 reslen:4194316 239ms Mon Feb 25 03:31:03 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:151159 nreturned:9458 reslen:4194389 151ms Mon Feb 25 03:31:03 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:159730 nreturned:9355 reslen:4194300 159ms Mon Feb 25 03:31:04 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:223034 nreturned:9462 reslen:4194683 223ms Mon Feb 25 03:31:04 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:118240 nreturned:9405 reslen:4194667 118ms Mon Feb 25 03:31:05 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:130425 nreturned:9410 reslen:4194508 130ms Mon Feb 25 03:31:05 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:206662 nreturned:9303 reslen:4194670 206ms Mon Feb 25 03:31:05 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:216010 nreturned:9412 reslen:4194713 216ms Mon Feb 25 03:31:05 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:103710 nreturned:9451 reslen:4194614 103ms Mon Feb 25 03:31:05 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:221718 nreturned:9465 reslen:4194507 113ms Mon Feb 25 03:31:06 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:169609 nreturned:9430 reslen:4194575 169ms Mon Feb 25 03:31:06 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:165353 nreturned:9355 reslen:4194499 165ms Mon Feb 25 03:31:06 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:159077 nreturned:9459 reslen:4194359 159ms Mon Feb 25 03:31:06 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:113328 nreturned:9448 reslen:4194716 113ms Mon Feb 25 03:31:06 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:208012 nreturned:9414 reslen:4194714 208ms Mon Feb 25 03:31:06 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:152801 nreturned:9467 reslen:4194531 152ms Mon Feb 25 03:31:07 [conn25590] getmore mlog.cartweb201302 query: { query: {}, $snapshot: true } cursorid:3917950144954227723 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:102999 nreturned:9447 reslen:4194682 103ms Mon Feb 25 03:31:12 [conn25391] serverStatus was very slow: { after basic: 0, middle of mem: 5380, after mem: 5380, after connections: 5380, after extra info: 5380, after counters: 5380, after repl: 5380, after asserts: 5380, after dur: 5380, at end: 5380 } Mon Feb 25 03:31:12 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:25 reslen:2824 5422ms Mon Feb 25 03:31:15 [conn25593] end connection 172.17.16.22:19358 (191 connections now open) Mon Feb 25 03:31:15 [initandlisten] connection accepted from 172.17.16.22:19360 #25600 (192 connections now open) Mon Feb 25 03:31:15 [conn25600] authenticate db: local { authenticate: 1, nonce: "97694919a62631a6", user: "__system", key: "569771bf845638b575ef781733197d96" } Mon Feb 25 03:31:15 [conn191] command mlog.$cmd command: { count: "cartweb201302", query: { now: { $gte: new Date(1361733960005), $lte: new Date(1361734260005) } } } ntoreturn:1 keyUpdates:0 numYields: 110 locks(micros) r:4633027 reslen:48 2366ms Mon Feb 25 03:31:18 [conn191] command mlog.$cmd command: { count: "cartweb201302", query: { now: { $gte: new Date(1361733960005), $lte: new Date(1361734260005) } } } ntoreturn:1 keyUpdates:0 numYields: 106 locks(micros) r:4220398 reslen:48 2267ms Mon Feb 25 03:31:20 [conn191] query mlog.cartweb201302 query: { now: { $gte: new Date(1361733960005), $lte: new Date(1361734260005) } } ntoreturn:500 ntoskip:0 nscanned:4553789 keyUpdates:0 numYields: 88 locks(micros) r:4714404 nreturned:29 reslen:26980 2382ms Mon Feb 25 03:31:26 [conn25598] end connection 172.17.16.23:25338 (191 connections now open) Mon Feb 25 03:31:26 [initandlisten] connection accepted from 172.17.16.23:25344 #25601 (192 connections now open) Mon Feb 25 03:31:26 [conn25601] authenticate db: local { authenticate: 1, nonce: "15b0996de660d9ae", user: "__system", key: "b4ffbe7fb392675d3d6e9991e2dd132d" } Mon Feb 25 03:31:27 [conn25590] getmore mlog.orderweb201302 query: { query: {}, $snapshot: true } cursorid:8582359807517800759 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:394165 nreturned:2270 reslen:4197600 394ms Mon Feb 25 03:31:27 [conn235] command mlog.$cmd command: { count: "order_cart20130225", query: { createDate: { $gte: new Date(1361733900000), $lte: new Date(1361734200000) } } } ntoreturn:1 keyUpdates:0 locks(micros) r:117065 reslen:48 117ms Mon Feb 25 03:31:27 [repl writer worker 1] build index mlog.tmp.mr.order_cart20130225_27795_inc { _id: 1 } Mon Feb 25 03:31:27 [repl writer worker 1] build index done. scanned 0 total records. 0 secs Mon Feb 25 03:31:27 [repl writer worker 1] info: creating collection mlog.tmp.mr.order_cart20130225_27795_inc on add index Mon Feb 25 03:31:27 [repl writer worker 1] build index mlog.tmp.mr.order_cart20130225_27795_inc { 0: 1 } Mon Feb 25 03:31:27 [repl writer worker 1] build index done. scanned 0 total records. 0 secs Mon Feb 25 03:31:27 [repl writer worker 1] build index mlog.tmp.mr.order_cart20130225_27795 { _id: 1 } Mon Feb 25 03:31:27 [repl writer worker 1] build index done. scanned 0 total records. 0 secs Mon Feb 25 03:31:27 [repl writer worker 1] CMD: drop mlog.tmp.mr.order_cart20130225_27795_inc Mon Feb 25 03:31:27 [repl writer worker 1] CMD: drop mlog.pinTopTemp_1361734273883 Mon Feb 25 03:31:28 [conn25590] getmore mlog.orderweb201302 query: { query: {}, $snapshot: true } cursorid:8582359807517800759 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:521275 nreturned:2202 reslen:4194561 404ms Mon Feb 25 03:31:28 [repl writer worker 1] build index mlog.tmp.mr.order_cart20130225_27796_inc { _id: 1 } Mon Feb 25 03:31:28 [repl writer worker 1] build index done. scanned 0 total records. 0 secs Mon Feb 25 03:31:28 [repl writer worker 1] info: creating collection mlog.tmp.mr.order_cart20130225_27796_inc on add index Mon Feb 25 03:31:28 [repl writer worker 1] build index mlog.tmp.mr.order_cart20130225_27796_inc { 0: 1 } Mon Feb 25 03:31:28 [repl writer worker 1] build index done. scanned 0 total records. 0 secs Mon Feb 25 03:31:28 [repl writer worker 1] build index mlog.tmp.mr.order_cart20130225_27796 { _id: 1 } Mon Feb 25 03:31:28 [repl writer worker 1] build index done. scanned 0 total records. 0 secs Mon Feb 25 03:31:28 [repl writer worker 1] CMD: drop mlog.tmp.mr.order_cart20130225_27796_inc Mon Feb 25 03:31:28 [repl writer worker 1] CMD: drop mlog.idOrder_1361734274066 Mon Feb 25 03:31:29 [conn25590] getmore mlog.orderweb201302 query: { query: {}, $snapshot: true } cursorid:8582359807517800759 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 2 locks(micros) r:1255987 nreturned:2204 reslen:4197014 888ms Mon Feb 25 03:31:29 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:78 reslen:2625 262ms Mon Feb 25 03:31:30 [conn25590] getmore mlog.orderweb201302 query: { query: {}, $snapshot: true } cursorid:8582359807517800759 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 6 locks(micros) r:1847265 nreturned:2299 reslen:4196797 925ms Mon Feb 25 03:31:31 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:41 reslen:2625 264ms Mon Feb 25 03:31:32 [conn25590] getmore mlog.orderweb201302 query: { query: {}, $snapshot: true } cursorid:8582359807517800759 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 8 locks(micros) r:3375856 nreturned:2052 reslen:4194977 2045ms Mon Feb 25 03:31:32 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:38 reslen:2625 259ms Mon Feb 25 03:31:33 [conn25590] getmore mlog.orderweb201302 query: { query: {}, $snapshot: true } cursorid:8582359807517800759 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 10 locks(micros) r:3402431 nreturned:2001 reslen:4194568 1701ms Mon Feb 25 03:31:33 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:28 reslen:2625 140ms Mon Feb 25 03:31:35 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:28 reslen:2625 595ms Mon Feb 25 03:31:35 [conn25590] getmore mlog.orderweb201302 query: { query: {}, $snapshot: true } cursorid:8582359807517800759 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:1528924 nreturned:2087 reslen:4194572 1528ms Mon Feb 25 03:31:44 [conn25590] getmore mlog.order_cart20130218 query: { query: {}, $snapshot: true } cursorid:9131450665979111236 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:375909 nreturned:1207 reslen:4195133 375ms Mon Feb 25 03:31:45 [conn25600] end connection 172.17.16.22:19360 (191 connections now open) Mon Feb 25 03:31:45 [initandlisten] connection accepted from 172.17.16.22:19364 #25602 (192 connections now open) Mon Feb 25 03:31:45 [conn25602] authenticate db: local { authenticate: 1, nonce: "8977e7c7a61c969a", user: "__system", key: "23ed287fb9373d57a670d8a695576d95" } Mon Feb 25 03:31:46 [conn25590] getmore mlog.order_cart20130218 query: { query: {}, $snapshot: true } cursorid:9131450665979111236 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:1032755 nreturned:1206 reslen:4196251 1032ms Mon Feb 25 03:31:47 [conn25590] getmore mlog.order_cart20130218 query: { query: {}, $snapshot: true } cursorid:9131450665979111236 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:957480 nreturned:1213 reslen:4196895 957ms Mon Feb 25 03:31:47 [rsSync] info DFM::findAll(): extent 30:76aad000 was empty, skipping ahead. ns:local.replset.minvalid Mon Feb 25 03:31:49 [conn25590] getmore mlog.order_cart20130218 query: { query: {}, $snapshot: true } cursorid:9131450665979111236 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:412069 nreturned:1192 reslen:4195828 412ms Mon Feb 25 03:31:50 [conn25590] getmore mlog.order_cart20130218 query: { query: {}, $snapshot: true } cursorid:9131450665979111236 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:428043 nreturned:1186 reslen:4195119 428ms Mon Feb 25 03:31:51 [conn25590] getmore mlog.order_cart20130219 query: { query: {}, $snapshot: true } cursorid:2475970039882226683 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:560515 nreturned:1183 reslen:4196682 560ms Mon Feb 25 03:31:52 [initandlisten] connection accepted from 192.168.137.100:43593 #25603 (193 connections now open) Mon Feb 25 03:31:52 [conn25603] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "8704901efd94e1fe", key: "a04f39ef496b46ac300c6dafbb905909" } Mon Feb 25 03:31:52 [conn25603] end connection 192.168.137.100:43593 (192 connections now open) Mon Feb 25 03:31:52 [initandlisten] connection accepted from 192.168.137.100:43595 #25604 (193 connections now open) Mon Feb 25 03:31:52 [conn25604] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "7a692bd538b5e450", key: "dc360413e9b996486cf2645f32600367" } Mon Feb 25 03:31:52 [conn25604] end connection 192.168.137.100:43595 (192 connections now open) Mon Feb 25 03:31:52 [conn25590] getmore mlog.order_cart20130219 query: { query: {}, $snapshot: true } cursorid:2475970039882226683 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 8 locks(micros) r:769482 nreturned:1257 reslen:4195683 386ms Mon Feb 25 03:31:52 [initandlisten] connection accepted from 192.168.137.100:43597 #25605 (193 connections now open) Mon Feb 25 03:31:52 [conn25605] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "b0960a26b686726a", key: "7e4655909f816d3eaeebf7e938074299" } Mon Feb 25 03:31:52 [conn25605] end connection 192.168.137.100:43597 (192 connections now open) Mon Feb 25 03:31:52 [initandlisten] connection accepted from 192.168.137.100:43598 #25606 (193 connections now open) Mon Feb 25 03:31:52 [conn25606] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "12bddccdaa233d71", key: "527f2371ef324323bd3d34dff9fa578e" } Mon Feb 25 03:31:52 [conn25606] end connection 192.168.137.100:43598 (192 connections now open) Mon Feb 25 03:31:53 [conn25590] getmore mlog.order_cart20130219 query: { query: {}, $snapshot: true } cursorid:2475970039882226683 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:739521 nreturned:1220 reslen:4197839 378ms Mon Feb 25 03:31:54 [conn25590] getmore mlog.order_cart20130219 query: { query: {}, $snapshot: true } cursorid:2475970039882226683 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:409876 nreturned:1246 reslen:4196734 409ms Mon Feb 25 03:31:55 [conn25590] getmore mlog.order_cart20130219 query: { query: {}, $snapshot: true } cursorid:2475970039882226683 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:393164 nreturned:1197 reslen:4194634 393ms Mon Feb 25 03:31:56 [conn25590] getmore mlog.order_cart20130219 query: { query: {}, $snapshot: true } cursorid:2475970039882226683 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 2 locks(micros) r:682066 nreturned:1202 reslen:4195810 405ms Mon Feb 25 03:31:56 [conn25601] end connection 172.17.16.23:25344 (191 connections now open) Mon Feb 25 03:31:56 [initandlisten] connection accepted from 172.17.16.23:25348 #25607 (192 connections now open) Mon Feb 25 03:31:56 [conn25607] authenticate db: local { authenticate: 1, nonce: "9cffa0b39bee297d", user: "__system", key: "852918690e3c5ed17408a05b57f2c19f" } Mon Feb 25 03:31:57 [conn25590] getmore mlog.order_cart20130219 query: { query: {}, $snapshot: true } cursorid:2475970039882226683 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:1752910 nreturned:1220 reslen:4195650 1152ms Mon Feb 25 03:31:58 [conn25590] getmore mlog.order_cart20130219 query: { query: {}, $snapshot: true } cursorid:2475970039882226683 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 2 locks(micros) r:1575934 nreturned:1209 reslen:4196152 1149ms Mon Feb 25 03:31:59 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:75 reslen:2625 239ms Mon Feb 25 03:32:00 [conn25590] getmore mlog.order_cart20130219 query: { query: {}, $snapshot: true } cursorid:2475970039882226683 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 5 locks(micros) r:2372331 nreturned:1227 reslen:4197562 1317ms Mon Feb 25 03:32:01 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:44 reslen:2625 103ms Mon Feb 25 03:32:01 [conn25590] getmore mlog.order_cart20130219 query: { query: {}, $snapshot: true } cursorid:2475970039882226683 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 5 locks(micros) r:2222683 nreturned:1229 reslen:4194751 1389ms Mon Feb 25 03:32:02 [initandlisten] connection accepted from 127.0.0.1:41138 #25608 (193 connections now open) Mon Feb 25 03:32:02 [conn25608] authenticate db: admin { authenticate: 1, nonce: "7147caeb8741912d", user: "admin", key: "b08817615b1534163dd80a6947d7ba04" } Mon Feb 25 03:32:02 [conn25608] authenticate db: admin { authenticate: 1, user: "admin", nonce: "7504e06ef36deb64", key: "4ce9ef6504f3eeb6d23461c43148359c" } Mon Feb 25 03:32:03 [conn25608] serverStatus was very slow: { after basic: 0, middle of mem: 1300, after mem: 1300, after connections: 1300, after extra info: 1300, after counters: 1300, after repl: 1300, after asserts: 1300, after dur: 1300, at end: 1300 } Mon Feb 25 03:32:03 [conn25608] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:25 reslen:2824 1307ms Mon Feb 25 03:32:03 [conn25608] end connection 127.0.0.1:41138 (192 connections now open) Mon Feb 25 03:32:12 [conn25590] getmore mlog.order_cart20130220 query: { query: {}, $snapshot: true } cursorid:6609479984693178096 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:329758 nreturned:1194 reslen:4196183 329ms Mon Feb 25 03:32:14 [conn561] command mlog.$cmd command: { count: "orderweb201302", query: { info.0: { $regex: /^.*exception.*$/i }, now: { $gte: new Date(1361734020004), $lte: new Date(1361734320004) }, $or: [ { level: 40000 } ] } } ntoreturn:1 keyUpdates:0 locks(micros) r:426705 reslen:48 426ms Mon Feb 25 03:32:14 [conn25590] getmore mlog.order_cart20130220 query: { query: {}, $snapshot: true } cursorid:6609479984693178096 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:1247056 nreturned:1273 reslen:4195165 624ms Mon Feb 25 03:32:14 [conn561] command mlog.$cmd command: { count: "orderweb201302", query: { info.0: { $regex: /^.*exception.*$/i }, now: { $gte: new Date(1361734020004), $lte: new Date(1361734320004) }, $or: [ { level: 40000 } ] } } ntoreturn:1 keyUpdates:0 locks(micros) r:228131 reslen:48 228ms Mon Feb 25 03:32:15 [conn25590] getmore mlog.order_cart20130220 query: { query: {}, $snapshot: true } cursorid:6609479984693178096 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 20 locks(micros) r:2660952 nreturned:1256 reslen:4197183 1341ms Mon Feb 25 03:32:15 [conn25602] end connection 172.17.16.22:19364 (191 connections now open) Mon Feb 25 03:32:15 [initandlisten] connection accepted from 172.17.16.22:19366 #25609 (192 connections now open) Mon Feb 25 03:32:15 [conn25609] authenticate db: local { authenticate: 1, nonce: "24bbbdb596fedb9", user: "__system", key: "5ad13dd79046c8a0246655420821e549" } Mon Feb 25 03:32:15 [conn561] query mlog.orderweb201302 query: { info.0: { $regex: /^.*exception.*$/i }, now: { $gte: new Date(1361734020004), $lte: new Date(1361734320004) }, $or: [ { level: 40000 } ] } ntoreturn:500 ntoskip:0 nscanned:386609 keyUpdates:0 numYields: 2 locks(micros) r:492451 nreturned:0 reslen:20 360ms Mon Feb 25 03:32:16 [conn25590] getmore mlog.order_cart20130220 query: { query: {}, $snapshot: true } cursorid:6609479984693178096 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 19 locks(micros) r:2598213 nreturned:1245 reslen:4197885 1352ms Mon Feb 25 03:32:17 [conn25590] getmore mlog.order_cart20130220 query: { query: {}, $snapshot: true } cursorid:6609479984693178096 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 5 locks(micros) r:2239402 nreturned:1237 reslen:4195726 1120ms Mon Feb 25 03:32:17 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:48 reslen:2645 198ms Mon Feb 25 03:32:18 [conn589] command mlog.$cmd command: { count: "cartweb201302", query: { info.0: { $regex: /^.*ConnectionLoss.*$/i }, now: { $gte: new Date(1361734020004), $lte: new Date(1361734320004) }, $or: [ { level: 40000 } ] } } ntoreturn:1 keyUpdates:0 numYields: 282 locks(micros) r:5231921 reslen:48 5159ms Mon Feb 25 03:32:19 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:37 reslen:2645 126ms Mon Feb 25 03:32:19 [conn25590] getmore mlog.order_cart20130220 query: { query: {}, $snapshot: true } cursorid:6609479984693178096 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 9 locks(micros) r:2240133 nreturned:1283 reslen:4194789 1393ms Mon Feb 25 03:32:20 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:60 reslen:2645 217ms Mon Feb 25 03:32:20 [conn25590] getmore mlog.order_cart20130220 query: { query: {}, $snapshot: true } cursorid:6609479984693178096 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 12 locks(micros) r:2653624 nreturned:1250 reslen:4199832 1328ms Mon Feb 25 03:32:21 [conn589] command mlog.$cmd command: { count: "cartweb201302", query: { info.0: { $regex: /^.*ConnectionLoss.*$/i }, now: { $gte: new Date(1361734020004), $lte: new Date(1361734320004) }, $or: [ { level: 40000 } ] } } ntoreturn:1 keyUpdates:0 numYields: 12 locks(micros) r:3978385 reslen:48 2959ms Mon Feb 25 03:32:24 [conn589] query mlog.cartweb201302 query: { info.0: { $regex: /^.*ConnectionLoss.*$/i }, now: { $gte: new Date(1361734020004), $lte: new Date(1361734320004) }, $or: [ { level: 40000 } ] } ntoreturn:500 ntoskip:0 nscanned:4553793 keyUpdates:0 numYields: 28 locks(micros) r:4979933 nreturned:0 reslen:20 2502ms Mon Feb 25 03:32:26 [conn25607] end connection 172.17.16.23:25348 (191 connections now open) Mon Feb 25 03:32:26 [initandlisten] connection accepted from 172.17.16.23:25354 #25610 (192 connections now open) Mon Feb 25 03:32:26 [conn25610] authenticate db: local { authenticate: 1, nonce: "c03844b186d9744a", user: "__system", key: "dce2090a315eb9844031bb92f2f6c1d0" } Mon Feb 25 03:32:31 [conn25590] getmore mlog.order_cart20130220 query: { query: {}, $snapshot: true } cursorid:6609479984693178096 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:423707 nreturned:1187 reslen:4195144 421ms Mon Feb 25 03:32:32 [conn25590] getmore mlog.order_cart20130221 query: { query: {}, $snapshot: true } cursorid:3154642492615375595 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:583804 nreturned:1188 reslen:4195037 583ms Mon Feb 25 03:32:33 [conn25590] getmore mlog.order_cart20130221 query: { query: {}, $snapshot: true } cursorid:3154642492615375595 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:512099 nreturned:1222 reslen:4195211 512ms Mon Feb 25 03:32:34 [conn25590] getmore mlog.order_cart20130221 query: { query: {}, $snapshot: true } cursorid:3154642492615375595 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:606734 nreturned:1246 reslen:4195736 497ms Mon Feb 25 03:32:35 [conn25590] getmore mlog.order_cart20130221 query: { query: {}, $snapshot: true } cursorid:3154642492615375595 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:464619 nreturned:1213 reslen:4197259 464ms Mon Feb 25 03:32:35 [conn25590] getmore mlog.order_cart20130221 query: { query: {}, $snapshot: true } cursorid:3154642492615375595 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:412235 nreturned:1258 reslen:4194578 412ms Mon Feb 25 03:32:36 [conn25590] getmore mlog.order_cart20130221 query: { query: {}, $snapshot: true } cursorid:3154642492615375595 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:396533 nreturned:1223 reslen:4194301 404ms Mon Feb 25 03:32:37 [conn25590] getmore mlog.order_cart20130221 query: { query: {}, $snapshot: true } cursorid:3154642492615375595 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:1021160 nreturned:1205 reslen:4194815 820ms Mon Feb 25 03:32:45 [conn25609] end connection 172.17.16.22:19366 (191 connections now open) Mon Feb 25 03:32:45 [initandlisten] connection accepted from 172.17.16.22:19370 #25611 (192 connections now open) Mon Feb 25 03:32:45 [conn25611] authenticate db: local { authenticate: 1, nonce: "a71de83d173a08f0", user: "__system", key: "b3de063de0d40e18f1a3ed16e50c8298" } Mon Feb 25 03:32:49 [rsSync] info DFM::findAll(): extent 30:76aad000 was empty, skipping ahead. ns:local.replset.minvalid Mon Feb 25 03:32:49 [initandlisten] connection accepted from 192.168.137.100:44868 #25612 (193 connections now open) Mon Feb 25 03:32:49 [conn25612] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "b28e986d07b6f94f", key: "38ea9b9096130464a8305ce5be14f39d" } Mon Feb 25 03:32:49 [conn25612] end connection 192.168.137.100:44868 (192 connections now open) Mon Feb 25 03:32:49 [initandlisten] connection accepted from 192.168.137.100:44869 #25613 (193 connections now open) Mon Feb 25 03:32:49 [conn25613] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "82d1e859facdf971", key: "92893fd8db3fbb9054c8b31a0f3c6a34" } Mon Feb 25 03:32:49 [conn25613] end connection 192.168.137.100:44869 (192 connections now open) Mon Feb 25 03:32:49 [initandlisten] connection accepted from 192.168.137.100:44870 #25614 (193 connections now open) Mon Feb 25 03:32:49 [conn25614] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "ea3849b34d3bef55", key: "379b172da67dc7da92b12f77028886a8" } Mon Feb 25 03:32:49 [conn25614] end connection 192.168.137.100:44870 (192 connections now open) Mon Feb 25 03:32:49 [initandlisten] connection accepted from 192.168.137.100:44871 #25615 (193 connections now open) Mon Feb 25 03:32:49 [conn25615] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "a63282bca33440a6", key: "084991690bf8f987edc3ad68939285d4" } Mon Feb 25 03:32:49 [conn25615] end connection 192.168.137.100:44871 (192 connections now open) Mon Feb 25 03:32:52 [conn25590] getmore mlog.order_cart20130222 query: { query: {}, $snapshot: true } cursorid:3059211218401200350 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:385128 nreturned:1188 reslen:4197339 384ms Mon Feb 25 03:32:53 [conn25590] getmore mlog.order_cart20130222 query: { query: {}, $snapshot: true } cursorid:3059211218401200350 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:428076 nreturned:1246 reslen:4194570 428ms Mon Feb 25 03:32:53 [conn25590] getmore mlog.order_cart20130222 query: { query: {}, $snapshot: true } cursorid:3059211218401200350 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:143743 nreturned:1223 reslen:4197139 143ms Mon Feb 25 03:32:55 [conn25590] getmore mlog.order_cart20130222 query: { query: {}, $snapshot: true } cursorid:3059211218401200350 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 2 locks(micros) r:1434926 nreturned:1223 reslen:4197163 1061ms Mon Feb 25 03:32:56 [conn25610] end connection 172.17.16.23:25354 (191 connections now open) Mon Feb 25 03:32:56 [initandlisten] connection accepted from 172.17.16.23:25358 #25616 (192 connections now open) Mon Feb 25 03:32:56 [conn25616] authenticate db: local { authenticate: 1, nonce: "a236b9de94abbe62", user: "__system", key: "1847b1321e3fe92e588e7c2379e76379" } Mon Feb 25 03:32:57 [conn25590] getmore mlog.order_cart20130222 query: { query: {}, $snapshot: true } cursorid:3059211218401200350 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:839109 nreturned:1200 reslen:4195159 424ms Mon Feb 25 03:32:57 [conn25590] getmore mlog.order_cart20130222 query: { query: {}, $snapshot: true } cursorid:3059211218401200350 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:291177 nreturned:1226 reslen:4196576 236ms Mon Feb 25 03:32:58 [conn25590] getmore mlog.order_cart20130222 query: { query: {}, $snapshot: true } cursorid:3059211218401200350 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:347969 nreturned:1202 reslen:4198089 175ms Mon Feb 25 03:32:58 [conn25590] getmore mlog.order_cart20130222 query: { query: {}, $snapshot: true } cursorid:3059211218401200350 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:770331 nreturned:1219 reslen:4195771 440ms Mon Feb 25 03:33:00 [conn25590] getmore mlog.order_cart20130222 query: { query: {}, $snapshot: true } cursorid:3059211218401200350 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 4 locks(micros) r:1675859 nreturned:1208 reslen:4194487 880ms Mon Feb 25 03:33:01 [conn25590] getmore mlog.order_cart20130222 query: { query: {}, $snapshot: true } cursorid:3059211218401200350 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:1374939 nreturned:1197 reslen:4197421 1374ms Mon Feb 25 03:33:02 [conn25590] getmore mlog.order_cart20130222 query: { query: {}, $snapshot: true } cursorid:3059211218401200350 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 2 locks(micros) r:736233 nreturned:1203 reslen:4195563 736ms Mon Feb 25 03:33:03 [conn25590] getmore mlog.order_cart20130222 query: { query: {}, $snapshot: true } cursorid:3059211218401200350 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 2 locks(micros) r:1005658 nreturned:1189 reslen:4195433 706ms Mon Feb 25 03:33:03 [conn25590] getmore mlog.order_cart20130222 query: { query: {}, $snapshot: true } cursorid:3059211218401200350 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:1376727 nreturned:1237 reslen:4196461 755ms Mon Feb 25 03:33:04 [initandlisten] connection accepted from 127.0.0.1:41149 #25617 (193 connections now open) Mon Feb 25 03:33:04 [conn25617] authenticate db: admin { authenticate: 1, nonce: "553fa1045da66372", user: "admin", key: "adb4b40a123391d60b36ad901cb0bbb6" } Mon Feb 25 03:33:04 [conn25617] authenticate db: admin { authenticate: 1, user: "admin", nonce: "6e681b94041a1d19", key: "9337cc49434defa532d203b72c96e354" } Mon Feb 25 03:33:04 [conn25617] command admin.$cmd command: { authenticate: 1, user: "admin", nonce: "6e681b94041a1d19", key: "9337cc49434defa532d203b72c96e354" } ntoreturn:1 keyUpdates:0 locks(micros) r:46 reslen:82 375ms Mon Feb 25 03:33:05 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:25 reslen:2625 206ms Mon Feb 25 03:33:05 [conn25590] getmore mlog.order_cart20130222 query: { query: {}, $snapshot: true } cursorid:3059211218401200350 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 24 locks(micros) r:2424757 nreturned:1190 reslen:4194932 1362ms Mon Feb 25 03:33:06 [conn25617] serverStatus was very slow: { after basic: 0, middle of mem: 0, after mem: 0, after connections: 0, after extra info: 0, after counters: 0, after repl: 0, after asserts: 0, after dur: 1770, at end: 1770 } Mon Feb 25 03:33:06 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:38 reslen:2633 601ms Mon Feb 25 03:33:06 [conn25617] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:61 reslen:2816 1784ms Mon Feb 25 03:33:06 [conn25617] end connection 127.0.0.1:41149 (192 connections now open) Mon Feb 25 03:33:15 [conn25611] end connection 172.17.16.22:19370 (191 connections now open) Mon Feb 25 03:33:15 [initandlisten] connection accepted from 172.17.16.22:19372 #25618 (192 connections now open) Mon Feb 25 03:33:15 [conn25618] authenticate db: local { authenticate: 1, nonce: "82db4649195bb2b1", user: "__system", key: "c7801d59e31f4086c89f29f587b8e102" } Mon Feb 25 03:33:15 [conn25590] ClientCursor::find(): cursor not found in map -1 (ok after a drop) getmore mlog.order_cart20130223 query: { query: {}, $snapshot: true } cursorid:6780784866963348692 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:258009 nreturned:1235 reslen:4194851 258ms Mon Feb 25 03:33:15 [conn25590] getmore mlog.order_cart20130223 query: { query: {}, $snapshot: true } cursorid:6780784866963348692 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:150273 nreturned:1214 reslen:4195899 150ms Mon Feb 25 03:33:16 [conn25590] getmore mlog.order_cart20130223 query: { query: {}, $snapshot: true } cursorid:6780784866963348692 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:313235 nreturned:1214 reslen:4197146 313ms Mon Feb 25 03:33:17 [conn270] command mlog.$cmd command: { count: "cartweb201302", query: { level: 40000, now: { $gte: new Date(1361734080004), $lte: new Date(1361734380004) }, $or: [ { info.0: { $regex: /^.*无法连接|connect.*$/i } } ] } } ntoreturn:1 keyUpdates:0 numYields: 679 locks(micros) r:6676795 reslen:48 3498ms Mon Feb 25 03:33:17 [conn25590] getmore mlog.order_cart20130223 query: { query: {}, $snapshot: true } cursorid:6780784866963348692 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:229460 nreturned:1217 reslen:4195747 229ms Mon Feb 25 03:33:17 [conn25590] getmore mlog.order_cart20130223 query: { query: {}, $snapshot: true } cursorid:6780784866963348692 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:235781 nreturned:1214 reslen:4195723 235ms Mon Feb 25 03:33:17 [conn25590] getmore mlog.order_cart20130223 query: { query: {}, $snapshot: true } cursorid:6780784866963348692 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:273706 nreturned:1216 reslen:4196277 171ms Mon Feb 25 03:33:17 [conn25590] getmore mlog.order_cart20130223 query: { query: {}, $snapshot: true } cursorid:6780784866963348692 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:195241 nreturned:1203 reslen:4196306 195ms Mon Feb 25 03:33:17 [conn25590] getmore mlog.order_cart20130223 query: { query: {}, $snapshot: true } cursorid:6780784866963348692 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 6 locks(micros) r:312917 nreturned:1229 reslen:4196543 180ms Mon Feb 25 03:33:18 [conn25590] getmore mlog.order_cart20130223 query: { query: {}, $snapshot: true } cursorid:6780784866963348692 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:214870 nreturned:1228 reslen:4195842 214ms Mon Feb 25 03:33:18 [conn25590] getmore mlog.order_cart20130223 query: { query: {}, $snapshot: true } cursorid:6780784866963348692 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:266941 nreturned:1210 reslen:4196825 266ms Mon Feb 25 03:33:18 [conn25590] getmore mlog.order_cart20130223 query: { query: {}, $snapshot: true } cursorid:6780784866963348692 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:321839 nreturned:1223 reslen:4196302 321ms Mon Feb 25 03:33:19 [conn25590] getmore mlog.order_cart20130223 query: { query: {}, $snapshot: true } cursorid:6780784866963348692 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 4 locks(micros) r:467976 nreturned:1208 reslen:4196179 275ms Mon Feb 25 03:33:19 [conn25590] getmore mlog.order_cart20130223 query: { query: {}, $snapshot: true } cursorid:6780784866963348692 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:260869 nreturned:1196 reslen:4196088 260ms Mon Feb 25 03:33:19 [conn25590] getmore mlog.order_cart20130223 query: { query: {}, $snapshot: true } cursorid:6780784866963348692 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:358746 nreturned:1204 reslen:4197428 358ms Mon Feb 25 03:33:19 [conn25590] getmore mlog.order_cart20130223 query: { query: {}, $snapshot: true } cursorid:6780784866963348692 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:314211 nreturned:1215 reslen:4194579 178ms Mon Feb 25 03:33:20 [conn25590] getmore mlog.order_cart20130223 query: { query: {}, $snapshot: true } cursorid:6780784866963348692 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 5 locks(micros) r:440722 nreturned:1185 reslen:4195600 221ms Mon Feb 25 03:33:24 [conn25391] serverStatus was very slow: { after basic: 0, middle of mem: 0, after mem: 0, after connections: 0, after extra info: 0, after counters: 0, after repl: 0, after asserts: 0, after dur: 3890, at end: 3890 } Mon Feb 25 03:33:24 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:30 reslen:2824 3922ms Mon Feb 25 03:33:25 [conn270] command mlog.$cmd command: { count: "cartweb201302", query: { level: 40000, now: { $gte: new Date(1361734080004), $lte: new Date(1361734380004) }, $or: [ { info.0: { $regex: /^.*无法连接|connect.*$/i } } ] } } ntoreturn:1 keyUpdates:0 numYields: 593 locks(micros) r:6487557 reslen:48 8266ms Mon Feb 25 03:33:26 [conn25616] end connection 172.17.16.23:25358 (191 connections now open) Mon Feb 25 03:33:26 [initandlisten] connection accepted from 172.17.16.23:25364 #25619 (192 connections now open) Mon Feb 25 03:33:26 [conn25619] authenticate db: local { authenticate: 1, nonce: "47b664d5e1e5e839", user: "__system", key: "6ac3dcf778fa8ebb21d8574558c9c92f" } Mon Feb 25 03:33:29 [conn270] query mlog.cartweb201302 query: { level: 40000, now: { $gte: new Date(1361734080004), $lte: new Date(1361734380004) }, $or: [ { info.0: { $regex: /^.*无法连接|connect.*$/i } } ] } ntoreturn:500 ntoskip:0 nscanned:4553813 keyUpdates:0 numYields: 160 locks(micros) r:7301226 nreturned:0 reslen:20 3689ms Mon Feb 25 03:33:34 [conn25590] getmore mlog.order_cart20130224 query: { query: {}, $snapshot: true } cursorid:228797001191533094 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:378597 nreturned:1219 reslen:4195263 240ms Mon Feb 25 03:33:35 [conn25590] getmore mlog.order_cart20130224 query: { query: {}, $snapshot: true } cursorid:228797001191533094 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:107235 nreturned:1243 reslen:4196450 107ms Mon Feb 25 03:33:35 [conn25590] getmore mlog.order_cart20130224 query: { query: {}, $snapshot: true } cursorid:228797001191533094 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:277139 nreturned:1271 reslen:4197540 277ms Mon Feb 25 03:33:35 [conn25590] getmore mlog.order_cart20130224 query: { query: {}, $snapshot: true } cursorid:228797001191533094 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:136872 nreturned:1240 reslen:4196972 136ms Mon Feb 25 03:33:36 [conn25590] getmore mlog.order_cart20130224 query: { query: {}, $snapshot: true } cursorid:228797001191533094 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:620110 nreturned:1232 reslen:4196599 620ms Mon Feb 25 03:33:36 [conn25590] getmore mlog.order_cart20130224 query: { query: {}, $snapshot: true } cursorid:228797001191533094 ntoreturn:0 exhaust:1 keyUpdates:0 numYields: 1 locks(micros) r:364111 nreturned:1237 reslen:4197664 285ms Mon Feb 25 03:33:39 [conn25590] getmore mlog.order_cart20130224 query: { query: {}, $snapshot: true } cursorid:228797001191533094 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:241583 nreturned:1219 reslen:4195563 241ms Mon Feb 25 03:33:39 [conn25590] getmore mlog.order_cart20130224 query: { query: {}, $snapshot: true } cursorid:228797001191533094 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:148073 nreturned:1203 reslen:4198444 148ms Mon Feb 25 03:33:40 [conn25590] getmore mlog.order_cart20130224 query: { query: {}, $snapshot: true } cursorid:228797001191533094 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:399316 nreturned:1178 reslen:4195890 399ms Mon Feb 25 03:33:40 [conn25590] getmore mlog.order_cart20130224 query: { query: {}, $snapshot: true } cursorid:228797001191533094 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:184332 nreturned:1215 reslen:4195581 184ms Mon Feb 25 03:33:40 [conn25590] getmore mlog.order_cart20130224 query: { query: {}, $snapshot: true } cursorid:228797001191533094 ntoreturn:0 exhaust:1 keyUpdates:0 locks(micros) r:191124 nreturned:1213 reslen:4198598 191ms Mon Feb 25 03:33:45 [conn25618] end connection 172.17.16.22:19372 (191 connections now open) Mon Feb 25 03:33:45 [initandlisten] connection accepted from 172.17.16.22:19376 #25620 (192 connections now open) Mon Feb 25 03:33:45 [conn25620] authenticate db: local { authenticate: 1, nonce: "4a88b101dad9d833", user: "__system", key: "eaf3754353e95a8e4a5f9ee06e5c0439" } Mon Feb 25 03:33:47 [TTLMonitor] query admin.system.indexes query: { expireAfterSeconds: { $exists: true } } ntoreturn:0 ntoskip:0 nscanned:1 keyUpdates:0 locks(micros) r:3453200 nreturned:0 reslen:20 3453ms Mon Feb 25 03:33:47 [conn25391] serverStatus was very slow: { after basic: 0, middle of mem: 6067, after mem: 6067, after connections: 6067, after extra info: 6067, after counters: 6067, after repl: 6067, after asserts: 6067, after dur: 6067, at end: 6067 } Mon Feb 25 03:33:47 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:54 reslen:2824 6109ms Mon Feb 25 03:33:47 [rsSync] info DFM::findAll(): extent 30:76aad000 was empty, skipping ahead. ns:local.replset.minvalid Mon Feb 25 03:33:56 [conn25619] end connection 172.17.16.23:25364 (191 connections now open) Mon Feb 25 03:33:56 [initandlisten] connection accepted from 172.17.16.23:25368 #25621 (192 connections now open) Mon Feb 25 03:33:56 [conn25621] authenticate db: local { authenticate: 1, nonce: "affb8f3ef6ffeace", user: "__system", key: "0b490ae881710fefd391fcf48811aadf" } Mon Feb 25 03:33:57 [initandlisten] connection accepted from 192.168.137.100:46605 #25622 (193 connections now open) Mon Feb 25 03:33:57 [conn25622] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "426dbe07c82d3031", key: "19d5760f2395fd26da89342add4657ff" } Mon Feb 25 03:33:57 [conn25622] end connection 192.168.137.100:46605 (192 connections now open) Mon Feb 25 03:33:57 [initandlisten] connection accepted from 192.168.137.100:46606 #25623 (193 connections now open) Mon Feb 25 03:33:57 [conn25623] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "c49638cfcd74fc82", key: "724d8b414d5b503dd9e23404cdd399d0" } Mon Feb 25 03:33:57 [conn25623] end connection 192.168.137.100:46606 (192 connections now open) Mon Feb 25 03:33:57 [initandlisten] connection accepted from 192.168.137.100:46607 #25624 (193 connections now open) Mon Feb 25 03:33:57 [conn25624] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "3ded7c76f3e8154e", key: "024f415f82fa2dfcf82e3605dfa03704" } Mon Feb 25 03:33:57 [conn25624] end connection 192.168.137.100:46607 (192 connections now open) Mon Feb 25 03:33:57 [initandlisten] connection accepted from 192.168.137.100:46608 #25625 (193 connections now open) Mon Feb 25 03:33:57 [conn25625] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "94b8dcf4ac3c5590", key: "3c4218ac022e96ba9a8b004778853d39" } Mon Feb 25 03:33:57 [conn25625] end connection 192.168.137.100:46608 (192 connections now open) Mon Feb 25 03:34:01 [initandlisten] connection accepted from 127.0.0.1:41158 #25626 (193 connections now open) Mon Feb 25 03:34:01 [conn25626] authenticate db: admin { authenticate: 1, nonce: "c39c4880df614163", user: "admin", key: "678408996051664e9f3068b468a5ff18" } Mon Feb 25 03:34:01 [conn25626] authenticate db: admin { authenticate: 1, user: "admin", nonce: "44f7df3337aa3d46", key: "17e6060b4665b8cc43acac99ff6e3237" } Mon Feb 25 03:34:01 [conn25626] end connection 127.0.0.1:41158 (192 connections now open) Mon Feb 25 03:34:13 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:46 reslen:2633 117ms Mon Feb 25 03:34:14 [conn25391] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:51 reslen:2633 134ms Mon Feb 25 03:34:15 [conn25620] end connection 172.17.16.22:19376 (191 connections now open) Mon Feb 25 03:34:15 [initandlisten] connection accepted from 172.17.16.22:19378 #25627 (192 connections now open) Mon Feb 25 03:34:15 [conn25627] authenticate db: local { authenticate: 1, nonce: "ab6d33942bff7b9a", user: "__system", key: "0583fb8c5f3157f56a848e4147916b85" } Mon Feb 25 03:34:16 [conn270] command mlog.$cmd command: { count: "cartweb201302", query: { info.0: { $regex: /^.*exception.*$/i }, now: { $gte: new Date(1361734140123), $lte: new Date(1361734440123) }, $or: [ { level: 40000 } ] } } ntoreturn:1 keyUpdates:0 numYields: 27093 locks(micros) r:4541880 reslen:48 2670ms Mon Feb 25 03:34:18 [conn270] command mlog.$cmd command: { count: "cartweb201302", query: { info.0: { $regex: /^.*exception.*$/i }, now: { $gte: new Date(1361734140123), $lte: new Date(1361734440123) }, $or: [ { level: 40000 } ] } } ntoreturn:1 keyUpdates:0 numYields: 7 locks(micros) r:3462818 reslen:48 2368ms Mon Feb 25 03:34:21 [conn270] query mlog.cartweb201302 query: { info.0: { $regex: /^.*exception.*$/i }, now: { $gte: new Date(1361734140123), $lte: new Date(1361734440123) }, $or: [ { level: 40000 } ] } ntoreturn:500 ntoskip:0 nscanned:4553817 keyUpdates:0 numYields: 14 locks(micros) r:4599715 nreturned:2 reslen:13739 2570ms Mon Feb 25 03:34:26 [conn25621] end connection 172.17.16.23:25368 (191 connections now open) Mon Feb 25 03:34:26 [initandlisten] connection accepted from 172.17.16.23:25374 #25628 (192 connections now open) Mon Feb 25 03:34:26 [conn25628] authenticate db: local { authenticate: 1, nonce: "85fc99806ae2e8e9", user: "__system", key: "805f2418dde1ec225c99c95df5d6b0f3" } Mon Feb 25 03:34:45 [conn25627] end connection 172.17.16.22:19378 (191 connections now open) Mon Feb 25 03:34:45 [initandlisten] connection accepted from 172.17.16.22:19382 #25629 (192 connections now open) Mon Feb 25 03:34:45 [conn25629] authenticate db: local { authenticate: 1, nonce: "c6a41bd013ad498a", user: "__system", key: "7540a72c20bb7fbf8fb0de49b04cd037" } Mon Feb 25 03:34:48 [initandlisten] connection accepted from 192.168.137.100:49815 #25630 (193 connections now open) Mon Feb 25 03:34:48 [conn25630] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "b925d4baa3ea34d4", key: "ed4c2c4d4ea1f143082d94659046d6a0" } Mon Feb 25 03:34:48 [conn25630] end connection 192.168.137.100:49815 (192 connections now open) Mon Feb 25 03:34:48 [initandlisten] connection accepted from 192.168.137.100:49819 #25631 (193 connections now open) Mon Feb 25 03:34:48 [conn25631] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "895006960c5cc644", key: "f6225c3a3d8e55b1089fbcbea99c00ab" } Mon Feb 25 03:34:48 [conn25631] end connection 192.168.137.100:49819 (192 connections now open) Mon Feb 25 03:34:49 [initandlisten] connection accepted from 192.168.137.100:49820 #25632 (193 connections now open) Mon Feb 25 03:34:49 [conn25632] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "2dae80044f1b7384", key: "6756bc3a86fea00d8ed2511b104e7948" } Mon Feb 25 03:34:49 [conn25632] end connection 192.168.137.100:49820 (192 connections now open) Mon Feb 25 03:34:49 [initandlisten] connection accepted from 192.168.137.100:49821 #25633 (193 connections now open) Mon Feb 25 03:34:49 [conn25633] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "d906a0d7dad90b7d", key: "52944dffb73a65638be5ddb7e15462e2" } Mon Feb 25 03:34:49 [conn25633] end connection 192.168.137.100:49821 (192 connections now open) Mon Feb 25 03:34:56 [conn25628] end connection 172.17.16.23:25374 (191 connections now open) Mon Feb 25 03:34:56 [initandlisten] connection accepted from 172.17.16.23:25378 #25634 (192 connections now open) Mon Feb 25 03:34:56 [conn25634] authenticate db: local { authenticate: 1, nonce: "65177f7e21a45a46", user: "__system", key: "7f89dbed3627c825dd20b2087889b766" } Mon Feb 25 03:35:01 [initandlisten] connection accepted from 127.0.0.1:41168 #25635 (193 connections now open) Mon Feb 25 03:35:01 [conn25635] authenticate db: admin { authenticate: 1, nonce: "80c4f7876ba0fd8e", user: "admin", key: "324fe85b032ea59bb71d94e369484c39" } Mon Feb 25 03:35:01 [conn25635] authenticate db: admin { authenticate: 1, user: "admin", nonce: "c34eadb93551a7f3", key: "f254154e3a0b61229dbafb7786f1c8a4" } Mon Feb 25 03:35:01 [conn25635] end connection 127.0.0.1:41168 (192 connections now open) Mon Feb 25 03:35:15 [conn25629] end connection 172.17.16.22:19382 (191 connections now open) Mon Feb 25 03:35:15 [initandlisten] connection accepted from 172.17.16.22:19384 #25636 (192 connections now open) Mon Feb 25 03:35:15 [conn25636] authenticate db: local { authenticate: 1, nonce: "e93e628224822da9", user: "__system", key: "c27ac2c74763474156351439ebe33715" } Mon Feb 25 03:35:16 [conn306] command mlog.$cmd command: { count: "cartweb201302", query: { info.0: { $regex: /^.*timeout.*$/i }, now: { $gte: new Date(1361734200003), $lte: new Date(1361734500003) }, $or: [ { level: 40000 } ] } } ntoreturn:1 keyUpdates:0 numYields: 178 locks(micros) r:5141447 reslen:48 2640ms Mon Feb 25 03:35:19 [conn306] command mlog.$cmd command: { count: "cartweb201302", query: { info.0: { $regex: /^.*timeout.*$/i }, now: { $gte: new Date(1361734200003), $lte: new Date(1361734500003) }, $or: [ { level: 40000 } ] } } ntoreturn:1 keyUpdates:0 numYields: 286 locks(micros) r:5000696 reslen:48 2750ms Mon Feb 25 03:35:21 [conn306] query mlog.cartweb201302 query: { info.0: { $regex: /^.*timeout.*$/i }, now: { $gte: new Date(1361734200003), $lte: new Date(1361734500003) }, $or: [ { level: 40000 } ] } ntoreturn:500 ntoskip:0 nscanned:4553818 keyUpdates:0 numYields: 2984 locks(micros) r:4951466 nreturned:0 reslen:20 2856ms Mon Feb 25 03:35:26 [conn25634] end connection 172.17.16.23:25378 (191 connections now open) Mon Feb 25 03:35:26 [initandlisten] connection accepted from 172.17.16.23:25384 #25637 (192 connections now open) Mon Feb 25 03:35:26 [conn25637] authenticate db: local { authenticate: 1, nonce: "586327c6b3ee4918", user: "__system", key: "44a0689ef480eda38aa86932d94b62e3" } Mon Feb 25 03:35:45 [conn25636] end connection 172.17.16.22:19384 (191 connections now open) Mon Feb 25 03:35:45 [initandlisten] connection accepted from 172.17.16.22:19388 #25638 (192 connections now open) Mon Feb 25 03:35:45 [conn25638] authenticate db: local { authenticate: 1, nonce: "1bf6ffcc50b51520", user: "__system", key: "19714301505447b3d23afb31e601a806" } Mon Feb 25 03:35:49 [initandlisten] connection accepted from 192.168.137.100:51097 #25639 (193 connections now open) Mon Feb 25 03:35:49 [conn25639] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "f9d0df399a338009", key: "464b54b71d82980bf080ea2bd09664cb" } Mon Feb 25 03:35:49 [conn25639] end connection 192.168.137.100:51097 (192 connections now open) Mon Feb 25 03:35:49 [initandlisten] connection accepted from 192.168.137.100:51098 #25640 (193 connections now open) Mon Feb 25 03:35:49 [conn25640] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "27825f759406683f", key: "b01ec0c576c856816a4374d5ab453815" } Mon Feb 25 03:35:49 [conn25640] end connection 192.168.137.100:51098 (192 connections now open) Mon Feb 25 03:35:49 [initandlisten] connection accepted from 192.168.137.100:51099 #25641 (193 connections now open) Mon Feb 25 03:35:49 [conn25641] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "a8319bd8fe6d50c1", key: "6b3e19407d40f62e5a369f65a5b141cf" } Mon Feb 25 03:35:49 [conn25641] end connection 192.168.137.100:51099 (192 connections now open) Mon Feb 25 03:35:49 [initandlisten] connection accepted from 192.168.137.100:51100 #25642 (193 connections now open) Mon Feb 25 03:35:49 [conn25642] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "fba42c9a9ec2ca44", key: "2b2755834c2d74e5b1f80d4b27eeb20e" } Mon Feb 25 03:35:49 [conn25642] end connection 192.168.137.100:51100 (192 connections now open) Mon Feb 25 03:35:56 [conn25637] end connection 172.17.16.23:25384 (191 connections now open) Mon Feb 25 03:35:56 [initandlisten] connection accepted from 172.17.16.23:25388 #25643 (192 connections now open) Mon Feb 25 03:35:56 [conn25643] authenticate db: local { authenticate: 1, nonce: "a0f73621641f195d", user: "__system", key: "6273af8a303c4d8c7fd3a135ea716e86" } Mon Feb 25 03:36:01 [initandlisten] connection accepted from 127.0.0.1:41178 #25644 (193 connections now open) Mon Feb 25 03:36:01 [conn25644] authenticate db: admin { authenticate: 1, nonce: "40578e38d0d5b7fd", user: "admin", key: "4645a19c1f8a8c051119a4422a49a044" } Mon Feb 25 03:36:01 [conn25644] authenticate db: admin { authenticate: 1, user: "admin", nonce: "1441a9344fa2d060", key: "aad3373c5274174cd7a4bffd2543bc36" } Mon Feb 25 03:36:01 [conn25644] end connection 127.0.0.1:41178 (192 connections now open) Mon Feb 25 03:36:15 [conn25638] end connection 172.17.16.22:19388 (191 connections now open) Mon Feb 25 03:36:15 [initandlisten] connection accepted from 172.17.16.22:19390 #25645 (192 connections now open) Mon Feb 25 03:36:15 [conn25645] authenticate db: local { authenticate: 1, nonce: "31951c0c7b4d18b8", user: "__system", key: "5861340cab847aec44d7cb9aa8a8a78f" } Mon Feb 25 03:36:16 [conn191] command mlog.$cmd command: { count: "cartweb201302", query: { now: { $gte: new Date(1361734260021), $lte: new Date(1361734560021) } } } ntoreturn:1 keyUpdates:0 numYields: 10 locks(micros) r:4682384 reslen:48 2842ms Mon Feb 25 03:36:19 [conn191] command mlog.$cmd command: { count: "cartweb201302", query: { now: { $gte: new Date(1361734260021), $lte: new Date(1361734560021) } } } ntoreturn:1 keyUpdates:0 numYields: 13 locks(micros) r:4696368 reslen:48 2699ms Mon Feb 25 03:36:22 [conn191] query mlog.cartweb201302 query: { now: { $gte: new Date(1361734260021), $lte: new Date(1361734560021) } } ntoreturn:500 ntoskip:0 nscanned:4553820 keyUpdates:0 numYields: 11 locks(micros) r:5269787 nreturned:31 reslen:57651 2934ms Mon Feb 25 03:36:26 [conn25643] end connection 172.17.16.23:25388 (191 connections now open) Mon Feb 25 03:36:26 [initandlisten] connection accepted from 172.17.16.23:25394 #25646 (192 connections now open) Mon Feb 25 03:36:26 [conn25646] authenticate db: local { authenticate: 1, nonce: "1c289ded464428c9", user: "__system", key: "36e239be84d59db83c0516b4a59bba44" } Mon Feb 25 03:36:28 [conn235] command mlog.$cmd command: { count: "order_cart20130225", query: { createDate: { $gte: new Date(1361734200000), $lte: new Date(1361734500000) } } } ntoreturn:1 keyUpdates:0 locks(micros) r:133112 reslen:48 133ms Mon Feb 25 03:36:45 [conn25645] end connection 172.17.16.22:19390 (191 connections now open) Mon Feb 25 03:36:45 [initandlisten] connection accepted from 172.17.16.22:19394 #25647 (192 connections now open) Mon Feb 25 03:36:45 [conn25647] authenticate db: local { authenticate: 1, nonce: "83a82c7858cc7efd", user: "__system", key: "c91fa8080410a42b5652c2fd4eb8a463" } Mon Feb 25 03:36:54 [initandlisten] connection accepted from 192.168.137.100:5865 #25648 (193 connections now open) Mon Feb 25 03:36:54 [conn25648] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "1f9765b3fbb43548", key: "93e19ac616bdcdd83f20fe8e2083d8bf" } Mon Feb 25 03:36:54 [conn25648] end connection 192.168.137.100:5865 (192 connections now open) Mon Feb 25 03:36:54 [initandlisten] connection accepted from 192.168.137.100:5866 #25649 (193 connections now open) Mon Feb 25 03:36:54 [conn25649] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "c0e10e2f66c3527b", key: "226a397fd6c958d9e4eaa6912a194823" } Mon Feb 25 03:36:54 [conn25649] end connection 192.168.137.100:5866 (192 connections now open) Mon Feb 25 03:36:54 [initandlisten] connection accepted from 192.168.137.100:5867 #25650 (193 connections now open) Mon Feb 25 03:36:54 [conn25650] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "b27c261c43daa987", key: "d8c867b651f5f580a3d316f9a64845e2" } Mon Feb 25 03:36:54 [conn25650] end connection 192.168.137.100:5867 (192 connections now open) Mon Feb 25 03:36:54 [initandlisten] connection accepted from 192.168.137.100:5868 #25651 (193 connections now open) Mon Feb 25 03:36:54 [conn25651] authenticate db: admin { authenticate: 1.0, user: "monitor", nonce: "497d66223cab810b", key: "0509f04ebb8124034f78e9dd905882b8" } Mon Feb 25 03:36:54 [conn25651] end connection 192.168.137.100:5868 (192 connections now open) Mon Feb 25 03:36:56 [conn25646] end connection 172.17.16.23:25394 (191 connections now open) Mon Feb 25 03:36:56 [initandlisten] connection accepted from 172.17.16.23:25398 #25652 (192 connections now open) Mon Feb 25 03:36:56 [conn25652] authenticate db: local { authenticate: 1, nonce: "3c68b6e01958730", user: "__system", key: "86de09e71b3d65568e0f12a97d34cf95" } Mon Feb 25 03:37:02 [initandlisten] connection accepted from 127.0.0.1:41188 #25653 (193 connections now open) Mon Feb 25 03:37:02 [conn25653] authenticate db: admin { authenticate: 1, nonce: "d88ede66ca1dc0f1", user: "admin", key: "5b3156a538272897199603774f91c5b7" } Mon Feb 25 03:37:02 [conn25653] authenticate db: admin { authenticate: 1, user: "admin", nonce: "8907efbfcafd0a33", key: "031dde9fc9e779368853a09860a2a07a" } Mon Feb 25 03:37:02 [conn25653] end connection 127.0.0.1:41188 (192 connections now open) Mon Feb 25 03:37:14 [conn191] command mlog.$cmd command: { count: "orderweb201302", query: { info.0: { $regex: /^.*exception.*$/i }, now: { $gte: new Date(1361734320005), $lte: new Date(1361734620005) }, $or: [ { level: 40000 } ] } } ntoreturn:1 keyUpdates:0 numYields: 530 locks(micros) r:849734 reslen:48 704ms Mon Feb 25 03:37:14 [conn191] command mlog.$cmd command: { count: "orderweb201302", query: { info.0: { $regex: /^.*exception.*$/i }, now: { $gte: new Date(1361734320005), $lte: new Date(1361734620005) }, $or: [ { level: 40000 } ] } } ntoreturn:1 keyUpdates:0 numYields: 471 locks(micros) r:323649 reslen:48 285ms Mon Feb 25 03:37:14 [repl writer worker 8] build index mlog.tmp.mr.order_cart20130225_27797_inc { _id: 1 } Mon Feb 25 03:37:14 [repl writer worker 8] build index done. scanned 0 total records. 0.006 secs Mon Feb 25 03:37:14 [repl writer worker 8] info: creating collection mlog.tmp.mr.order_cart20130225_27797_inc on add index Mon Feb 25 03:37:14 [repl writer worker 8] build index mlog.tmp.mr.order_cart20130225_27797_inc { 0: 1 } Mon Feb 25 03:37:14 [repl writer worker 8] build index done. scanned 0 total records. 0 secs Mon Feb 25 03:37:14 [repl writer worker 8] build index mlog.tmp.mr.order_cart20130225_27797 { _id: 1 } Mon Feb 25 03:37:14 [repl writer worker 8] build index done. scanned 0 total records. 0 secs Mon Feb 25 03:37:14 [conn191] query mlog.orderweb201302 query: { info.0: { $regex: /^.*exception.*$/i }, now: { $gte: new Date(1361734320005), $lte: new Date(1361734620005) }, $or: [ { level: 40000 } ] } ntoreturn:500 ntoskip:0 nscanned:386609 keyUpdates:0 numYields: 687 locks(micros) r:531745 nreturned:0 reslen:20 304ms Mon Feb 25 03:37:15 [conn25647] end connection 172.17.16.22:19394 (191 connections now open) Mon Feb 25 03:37:15 [initandlisten] connection accepted from 172.17.16.22:19396 #25654 (192 connections now open) Mon Feb 25 03:37:15 [conn25654] authenticate db: local { authenticate: 1, nonce: "397f1dc02ad412f1", user: "__system", key: "d65451e5383f30d81dd3d5d5b1c50d55" } Mon Feb 25 03:37:16 [conn232] command mlog.$cmd command: { count: "cartweb201302", query: { info.0: { $regex: /^.*ConnectionLoss.*$/i }, now: { $gte: new Date(1361734320005), $lte: new Date(1361734620005) }, $or: [ { level: 40000 } ] } } ntoreturn:1 keyUpdates:0 numYields: 3603 locks(micros) r:4708931 reslen:48 2709ms Mon Feb 25 03:37:16 [repl writer worker 8] CMD: drop mlog.tmp.mr.order_cart20130225_27797_inc Mon Feb 25 03:37:16 [repl writer worker 8] CMD: drop mlog.pinTopTemp_1361734574452 Mon Feb 25 03:37:17 [repl writer worker 8] build index mlog.tmp.mr.order_cart20130225_27798_inc { _id: 1 } Mon Feb 25 03:37:17 [repl writer worker 8] build index done. scanned 0 total records. 0 secs Mon Feb 25 03:37:17 [repl writer worker 8] info: creating collection mlog.tmp.mr.order_cart20130225_27798_inc on add index Mon Feb 25 03:37:17 [repl writer worker 8] build index mlog.tmp.mr.order_cart20130225_27798_inc { 0: 1 } Mon Feb 25 03:37:17 [repl writer worker 8] build index done. scanned 0 total records. 0 secs Mon Feb 25 03:37:18 [rsSync] info DFM::findAll(): extent 30:76aad000 was empty, skipping ahead. ns:local.replset.minvalid Mon Feb 25 03:37:18 [repl writer worker 8] build index mlog.tmp.mr.order_cart20130225_27798 { _id: 1 } Mon Feb 25 03:37:18 [repl writer worker 8] build index done. scanned 0 total records. 0.034 secs Mon Feb 25 03:37:19 [conn232] command mlog.$cmd command: { count: "cartweb201302", query: { info.0: { $regex: /^.*ConnectionLoss.*$/i }, now: { $gte: new Date(1361734320005), $lte: new Date(1361734620005) }, $or: [ { level: 40000 } ] } } ntoreturn:1 keyUpdates:0 numYields: 2366 locks(micros) r:5339198 reslen:48 3036ms Mon Feb 25 03:37:20 [repl writer worker 8] CMD: drop mlog.tmp.mr.order_cart20130225_27798_inc Mon Feb 25 03:37:21 [repl writer worker 8] CMD: drop mlog.idOrder_1361734574590 Mon Feb 25 03:37:22 [conn232] query mlog.cartweb201302 query: { info.0: { $regex: /^.*ConnectionLoss.*$/i }, now: { $gte: new Date(1361734320005), $lte: new Date(1361734620005) }, $or: [ { level: 40000 } ] } ntoreturn:500 ntoskip:0 nscanned:4553822 keyUpdates:0 numYields: 1784 locks(micros) r:5709404 nreturned:0 reslen:20 3063ms Mon Feb 25 03:37:26 [conn25652] end connection 172.17.16.23:25398 (191 connections now open) Mon Feb 25 03:37:26 [initandlisten] connection accepted from 172.17.16.23:25404 #25655 (193 connections now open) Mon Feb 25 03:37:26 [conn25655] authenticate db: local { authenticate: 1, nonce: "68912330446e7d65", user: "__system", key: "df549309fa7d511ce6fe453faffc50e8" } Mon Feb 25 03:37:45 [conn25654] end connection 172.17.16.22:19396 (191 connections now open) Mon Feb 25 03:37:45 [initandlisten] connection accepted from 172.17.16.22:19400 #25656 (192 connections now open) Mon Feb 25 03:37:45 [conn25656] authenticate db: local { authenticate: 1, nonce: "4263a1a006939b82", user: "__system", key: "aea96e6554209bd7f6b321699ca318ca" } Mon Feb 25 03:37:46 [conn25590] query local.oplog.rs query: { ts: { $gt: Timestamp 1361734158000|1 } } cursorid:7575054835460212505 ntoreturn:0 ntoskip:0 exhaust:1 nscanned:79628572 keyUpdates:0 numYields: 4926 locks(micros) r:460797038 nreturned:101 reslen:329421 231733ms Mon Feb 25 03:37:46 Invalid access at address: 0 from thread: conn25391 Mon Feb 25 03:37:46 Got signal: 7 (Bus error). Mon Feb 25 03:37:46 Backtrace: 0xade6e1 0x5582d9 0x558862 0x34a320f4a0 0x9b06f3 0xc22872 0x5c99df 0x66b636 0x6571f1 0x658d5d 0x659c1e 0x6afa08 0x6b3241 0xb5ba7d 0xb5d052 0x56fa52 0x5dbf11 0x34a32077f1 0x34a2ee570d /export/servers/mongodb/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /export/servers/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5582d9] /export/servers/mongodb/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262) [0x558862] /lib64/libpthread.so.0() [0x34a320f4a0] /export/servers/mongodb/bin/mongod() [0x9b06f3] /export/servers/mongodb/bin/mongod(malloc+0xe2) [0xc22872] /export/servers/mongodb/bin/mongod(_ZN5mongo14prettyHostNameEv+0x1f) [0x5c99df] /export/servers/mongodb/bin/mongod(_ZN5mongo15CmdServerStatus3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xc6) [0x66b636] /export/servers/mongodb/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6571f1] /export/servers/mongodb/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xcfd) [0x658d5d] /export/servers/mongodb/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ae) [0x659c1e] /export/servers/mongodb/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x38) [0x6afa08] /export/servers/mongodb/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xbc1) [0x6b3241] /export/servers/mongodb/bin/mongod() [0xb5ba7d] /export/servers/mongodb/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3a2) [0xb5d052] /export/servers/mongodb/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x56fa52] /export/servers/mongodb/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0x5dbf11] /lib64/libpthread.so.0() [0x34a32077f1] /lib64/libc.so.6(clone+0x6d) [0x34a2ee570d] ***** SERVER RESTARTED ***** Mon Feb 25 09:14:18 [initandlisten] MongoDB starting : pid=26045 port=20011 dbpath=/export/data/mongodb/mongod 64-bit host=orderlog2 Mon Feb 25 09:14:18 [initandlisten] db version v2.2.0, pdfile version 4.5 Mon Feb 25 09:14:18 [initandlisten] git version: f5e83eae9cfbec7fb7a071321928f00d1b0c5207 Mon Feb 25 09:14:18 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49 Mon Feb 25 09:14:18 [initandlisten] options: { config: "/export/servers/mongodb/etc/mongod.cnf", dbpath: "/export/data/mongodb/mongod", directoryperdb: "true", fork: "true", keyFile: "/export/data/mongodb/key/key", logappend: "true", logpath: "/export/data/mongodb/log/mongod.log", maxConns: 12000, pidfilepath: "/export/data/mongodb/pid/mongod.pid", port: 20011, profile: 1, replSet: "orderlog", rest: "true" } Mon Feb 25 09:14:18 [initandlisten] journal dir=/export/data/mongodb/mongod/journal Mon Feb 25 09:14:18 [initandlisten] recover begin Mon Feb 25 09:14:18 [initandlisten] recover lsn: 167889109 Mon Feb 25 09:14:18 [initandlisten] recover /export/data/mongodb/mongod/journal/j._8 Mon Feb 25 09:14:18 [initandlisten] recover skipping application of section seq:166342374 < lsn:167889109 Mon Feb 25 09:14:18 [initandlisten] recover skipping application of section seq:166401842 < lsn:167889109 Mon Feb 25 09:14:18 [initandlisten] recover skipping application of section seq:166461296 < lsn:167889109 Mon Feb 25 09:14:18 [initandlisten] recover skipping application of section seq:166520762 < lsn:167889109 Mon Feb 25 09:14:18 [initandlisten] recover skipping application of section seq:166580219 < lsn:167889109 Mon Feb 25 09:14:18 [initandlisten] recover skipping application of section seq:166639682 < lsn:167889109 Mon Feb 25 09:14:18 [initandlisten] recover skipping application of section seq:166699140 < lsn:167889109 Mon Feb 25 09:14:18 [initandlisten] recover skipping application of section seq:166758614 < lsn:167889109 Mon Feb 25 09:14:18 [initandlisten] recover skipping application of section seq:166818083 < lsn:167889109 Mon Feb 25 09:14:18 [initandlisten] recover skipping application of section more... Mon Feb 25 09:14:18 [initandlisten] recover cleaning up Mon Feb 25 09:14:18 [initandlisten] removeJournalFiles Mon Feb 25 09:14:19 [initandlisten] recover done Mon Feb 25 09:14:19 [initandlisten] preallocating a journal file /export/data/mongodb/mongod/journal/prealloc.0 Mon Feb 25 09:14:22 [websvr] admin web console waiting for connections on port 21011 Mon Feb 25 09:14:22 [initandlisten] waiting for connections on port 20011 Mon Feb 25 09:14:22 [initandlisten] connection accepted from 172.17.16.23:47945 #1 (1 connection now open) Mon Feb 25 09:14:22 [conn1] authenticate db: local { authenticate: 1, nonce: "c5e69e63b085625f", user: "__system", key: "78f0344a18e6a9df83afd58efe7912ee" } Mon Feb 25 09:14:23 [initandlisten] connection accepted from 192.168.132.105:55437 #2 (2 connections now open) Mon Feb 25 09:14:23 [initandlisten] connection accepted from 172.17.23.30:56075 #3 (3 connections now open) Mon Feb 25 09:14:23 [initandlisten] connection accepted from 192.168.129.172:20918 #4 (4 connections now open) Mon Feb 25 09:14:23 [initandlisten] connection accepted from 172.17.3.134:55667 #5 (5 connections now open) Mon Feb 25 09:14:23 [conn5] authenticate db: admin { authenticate: 1, nonce: "62515ca0694c05d3", user: "monitor", key: "6c65189ea930f5977398e732107c91b0" } Mon Feb 25 09:14:23 [initandlisten] connection accepted from 192.168.132.109:31358 #6 (6 connections now open) Mon Feb 25 09:14:23 [initandlisten] connection accepted from 192.168.132.105:55445 #7 (7 connections now open) Mon Feb 25 09:14:23 [initandlisten] connection accepted from 172.17.23.30:56078 #8 (8 connections now open) Mon Feb 25 09:14:23 [initandlisten] connection accepted from 172.17.23.29:30791 #9 (9 connections now open) Mon Feb 25 09:14:23 [initandlisten] connection accepted from 192.168.130.109:5152 #10 (10 connections now open) Mon Feb 25 09:14:23 [initandlisten] connection accepted from 192.168.130.109:5153 #11 (11 connections now open) Mon Feb 25 09:14:23 [rsStart] replSet I am 172.17.16.23:20011 Mon Feb 25 09:14:23 [rsStart] replSet STARTUP2 Mon Feb 25 09:14:23 [rsHealthPoll] replSet member 172.17.16.23:20012 is up Mon Feb 25 09:14:23 [rsHealthPoll] replSet member 172.17.16.23:20012 is now in state ARBITER Mon Feb 25 09:14:23 [rsHealthPoll] replSet member 172.17.16.22:20011 is up Mon Feb 25 09:14:23 [rsHealthPoll] replSet member 172.17.16.22:20011 is now in state PRIMARY