[SERVER-4573] faults with every action Created: 28/Dec/11  Updated: 06/Apr/23  Resolved: 08/Feb/12

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: 2.0.2
Fix Version/s: None

Type: Question Priority: Minor - P4
Reporter: Brett Goldstein Assignee: Tad Marshall
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

RH Enterprise dual proc server


Participants:

 Description   

Running a lab environment with approx 40 million documents. Watching mongostat, every action taken is incurring faults. What would be your recommended technique to identify the causality.



 Comments   
Comment by Tad Marshall [ 31/Jan/12 ]

From the numbers in your mongostat output, it seems that your mapped dataset size (9.95 GB) is a lot larger than the amount that actually fits in memory (between 1.88 GB and 2.74 GB), so page faults will happen for every document that falls outside the resident memory. As expected, the fault rate is higher for inserts than for deletes (since a record to be deleted might be in memory, while an insert requires free space in a file region that probably won't be in memory if you are memory-constrained). Since you probably have no control over the locality of the data being removed or inserted, your best bet would be to increase RAM.

Comment by Eliot Horowitz (Inactive) [ 29/Dec/11 ]

If the data set is a lot bigger than ram - than if operations are evenly distributed - most operations will produce a fault.

Comment by Brett Goldstein [ 29/Dec/11 ]

this is from an insert job:

[mongodb@my07apl01 ~]$ ./mongodb-linux-x86_64-2.0.2/bin/mongostat
connected to: 127.0.0.1
insert query update delete getmore command flushes mapped vsize res faults locked % idx miss % qr|qw ar|aw netIn netOut conn time
228 0 0 0 0 1 0 9.95g 20.3g 2.52g 290 95.6 2.9 0|0 0|1 78k 1k 5 18:47:36
245 0 0 0 0 1 0 9.95g 20.3g 2.5g 257 96.7 2.7 0|0 0|1 84k 1k 5 18:47:37
175 0 0 0 0 1 0 9.95g 20.3g 2.54g 308 97.7 7.4 0|0 0|1 59k 1k 5 18:47:38
157 0 0 0 0 1 0 9.95g 20.3g 2.57g 230 95.4 4.1 0|0 0|1 55k 1k 5 18:47:39
210 0 0 0 0 1 0 9.95g 20.3g 2.55g 161 94.5 6.2 0|0 0|1 75k 1k 5 18:47:40
45 0 0 0 0 1 0 9.95g 20.3g 2.56g 77 98.8 0 0|0 0|1 15k 1k 5 18:47:41
107 0 0 0 0 1 1 9.95g 20.3g 2.56g 128 98.5 0 0|0 0|1 37k 1k 5 18:47:42
235 0 0 0 0 1 0 9.95g 20.3g 2.61g 309 97.2 5.4 0|0 0|1 85k 1k 5 18:47:43
244 0 0 0 0 1 0 9.95g 20.3g 2.62g 304 96.1 2.7 0|0 0|1 83k 1k 5 18:47:44
207 0 0 0 0 1 0 9.95g 20.3g 2.65g 327 97.7 3.2 0|0 0|0 71k 1k 5 18:47:45
insert query update delete getmore command flushes mapped vsize res faults locked % idx miss % qr|qw ar|aw netIn netOut conn time
216 0 0 0 0 1 0 9.95g 20.3g 2.63g 292 95.1 3 0|0 0|1 78k 1k 5 18:47:46
282 0 0 0 0 1 0 9.95g 20.3g 2.68g 359 97.9 2.3 0|0 0|1 98k 1k 5 18:47:47
294 0 0 0 0 1 0 9.95g 20.3g 2.69g 335 96.5 4.4 0|0 0|1 102k 1k 5 18:47:48
305 0 0 0 0 1 0 9.95g 20.3g 2.73g 243 93.4 6.5 0|0 0|1 105k 1k 5 18:47:49
353 0 0 0 0 1 0 9.95g 20.3g 2.76g 261 97.7 1.8 0|0 0|1 121k 1k 5 18:47:50
247 0 0 0 0 1 0 9.95g 20.3g 2.73g 196 95.5 5.4 0|0 0|1 85k 1k 5 18:47:51
378 0 0 0 0 1 0 9.95g 20.3g 2.74g 320 95.4 0 0|0 0|1 137k 1k 5 18:47:52

[mongodb@my07apl01 ~]$

Comment by Brett Goldstein [ 29/Dec/11 ]

This is from a large remove:

[mongodb@my07apl01 ~]$ ./mongodb-linux-x86_64-2.0.2/bin/mongostat
connected to: 127.0.0.1
insert query update delete getmore command flushes mapped vsize res faults locked % idx miss % qr|qw ar|aw netIn netOut conn time
0 0 0 0 0 1 0 9.95g 20.3g 1.9g 0 84.5 0 0|0 0|1 62b 1k 4 18:32:17
0 0 0 0 0 1 0 9.95g 20.3g 1.88g 11 136 0 0|0 0|1 62b 1k 4 18:32:18
0 0 0 0 0 1 0 9.95g 20.3g 1.9g 0 89.4 0 0|0 0|1 62b 1k 4 18:32:19
0 0 0 0 0 1 0 9.95g 20.3g 1.9g 0 85.6 0 0|0 0|1 62b 1k 4 18:32:20
0 0 0 0 0 1 0 9.95g 20.3g 1.9g 0 86.7 0 0|0 0|1 62b 1k 4 18:32:21
0 0 0 0 0 1 0 9.95g 20.3g 1.9g 0 130 0 0|0 0|1 62b 1k 4 18:32:22
0 0 0 0 0 1 0 9.95g 20.3g 1.9g 6 80.1 0 0|0 0|1 62b 1k 4 18:32:23
0 0 0 0 0 1 0 9.95g 20.3g 1.9g 0 80.6 0 0|0 0|1 62b 1k 4 18:32:24
0 0 0 0 0 1 0 9.95g 20.3g 1.91g 1 77.1 0 0|0 0|1 62b 1k 4 18:32:25
0 0 0 0 0 1 0 9.95g 20.3g 1.9g 18 164 0 0|0 0|1 62b 1k 4 18:32:26
insert query update delete getmore command flushes mapped vsize res faults locked % idx miss % qr|qw ar|aw netIn netOut conn time
0 0 0 0 0 1 0 9.95g 20.3g 1.91g 1 86.6 0 0|0 0|1 62b 1k 4 18:32:27
0 0 0 0 0 1 0 9.95g 20.3g 1.93g 0 86.6 0 0|0 0|1 62b 1k 4 18:32:28
0 0 0 0 0 1 0 9.95g 20.3g 1.89g 0 88.7 0 0|0 0|1 62b 1k 4 18:32:29
0 0 0 0 0 1 0 9.95g 20.3g 1.92g 0 54.9 0 0|1 0|1 62b 1k 4 18:32:30
0 0 0 0 0 1 0 9.95g 20.3g 1.92g 0 175 0 0|0 0|1 62b 1k 4 18:32:31
0 0 0 0 0 1 0 9.95g 20.3g 1.92g 0 85.1 0 0|0 0|1 62b 1k 4 18:32:32
0 0 0 0 0 1 0 9.95g 20.3g 1.89g 1 83.5 0 0|0 0|1 62b 1k 4 18:32:33
0 0 0 0 0 1 0 9.95g 20.3g 1.92g 0 53.8 0 0|1 0|1 62b 1k 4 18:32:34
0 0 0 0 0 1 0 9.95g 20.3g 1.92g 11 156 0 0|0 0|1 62b 1k 4 18:32:35
0 0 0 0 0 1 0 9.95g 20.3g 1.91g 0 73.2 0 0|0 0|1 62b 1k 4 18:32:36
insert query update delete getmore command flushes mapped vsize res faults locked % idx miss % qr|qw ar|aw netIn netOut conn time
0 0 0 0 0 1 0 9.95g 20.3g 1.92g 0 74.9 0 0|1 0|1 62b 1k 4 18:32:37

Comment by Eliot Horowitz (Inactive) [ 28/Dec/11 ]

Can you paste mongostat output?

Generated at Thu Feb 08 03:06:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.