[SERVER-20728] mongod hang during 3-node replSet performance test with 3.0.7-pre (SHA: 180db3d2) Created: 01/Oct/15  Updated: 13/Oct/15  Resolved: 02/Oct/15

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.0.7
Fix Version/s: 3.0.7

Type: Bug Priority: Major - P3
Reporter: Rui Zhang (Inactive) Assignee: Michael Cahill (Inactive)
Resolution: Done Votes: 0
Labels: sys-perf-reg
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File stack-trace.log    
Issue Links:
Related
is related to SERVER-16902 wt cache: maximum page size at evicti... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

mongod hang during EVG 3.0.7-pre patch testing

Setup/Test Details

Observation

  • mongod hang during YCSB 50 read/50 update test
  • CPU usage on the primary is low
  • shell will hang if do insert/find
  • looked at pending request, lots of “aw” from mongostat

    [ec2-user@ip-10-2-0-100 ~]$ ./mongodb-linux-x86_64-3.0.7-pre-/bin/mongostat
    insert query update delete getmore command % dirty % used flushes vsize   res qr|qw ar|aw netIn netOut conn set repl     time
        *0    *0     *0     *0       0     5|0    35.7   40.8       0 13.2G 12.6G   1|7  1|58  414b    63k   76 rs0  PRI 22:17:29
        *0    *0     *0     *0       0     6|0    35.7   40.8       0 13.2G 12.6G   1|7  1|58  472b    63k   76 rs0  PRI 22:17:30
        *0    *0     *0     *0       0     5|0    35.7   40.8       0 13.2G 12.6G   1|6  1|59  414b    63k   76 rs0  PRI 22:17:31
    

  • output from YCSB

     [2015/10/01 18:00:24.904]  100 sec: 7926293 operations; 88118.59 current ops/sec; [UPDATE AverageLatency(us)=1036.89] [READ AverageLatency(us)=406.35]
     [2015/10/01 18:00:34.897]  110 sec: 8724240 operations; 79794.7 current ops/sec; [UPDATE AverageLatency(us)=1158.45] [READ AverageLatency(us)=432.6]
     [2015/10/01 18:00:44.897]  120 sec: 9584935 operations; 86060.89 current ops/sec; [UPDATE AverageLatency(us)=1062.74] [READ AverageLatency(us)=416.89]
     [2015/10/01 18:00:54.898]  130 sec: 10387888 operations; 80287.27 current ops/sec; [UPDATE AverageLatency(us)=1138.84] [READ AverageLatency(us)=445.68]
     [2015/10/01 18:01:04.898]  140 sec: 11229191 operations; 84130.3 current ops/sec; [UPDATE AverageLatency(us)=1097.95] [READ AverageLatency(us)=413.37]
     [2015/10/01 18:01:14.899]  150 sec: 12017060 operations; 78779.02 current ops/sec; [UPDATE AverageLatency(us)=1155.19] [READ AverageLatency(us)=462.05]
     [2015/10/01 18:01:24.899]  160 sec: 12087119 operations; 7005.9 current ops/sec; [UPDATE AverageLatency(us)=1683.19] [READ AverageLatency(us)=535.9]
     [2015/10/01 18:01:34.899]  170 sec: 12087119 operations; 0 current ops/sec;
     [2015/10/01 18:01:44.899]  180 sec: 12087119 operations; 0 current ops/sec;
     [2015/10/01 18:01:54.899]  190 sec: 12087119 operations; 0 current ops/sec;
     [2015/10/01 18:02:04.899]  200 sec: 12087119 operations; 0 current ops/sec;
     [2015/10/01 18:02:14.899]  210 sec: 12087119 operations; 0 current ops/sec;
     [2015/10/01 18:02:24.899]  220 sec: 12087119 operations; 0 current ops/sec;
     [2015/10/01 18:02:34.900]  230 sec: 12087119 operations; 0 current ops/sec;
     [2015/10/01 18:02:44.900]  240 sec: 12087119 operations; 0 current ops/sec;
     [2015/10/01 18:02:54.900]  250 sec: 12087119 operations; 0 current ops/sec;
     [2015/10/01 18:03:04.901]  260 sec: 12087119 operations; 0 current ops/sec;
     [2015/10/01 18:03:14.901]  270 sec: 12087119 operations; 0 current ops/sec;
    
    



 Comments   
Comment by Michael Cahill (Inactive) [ 02/Oct/15 ]

This should be resolved by https://github.com/mongodb/mongo/commit/3b865f94cbdc1b6c71f6233ac6675e5edfa34ebd

Comment by Michael Cahill (Inactive) [ 02/Oct/15 ]

rui.zhang, I've got everything I need, thanks – I'm testing a change now. Feel free to shut down the test.

Generated at Thu Feb 08 03:55:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.