[SERVER-16064] WT Performance Regression for update of hot document on 2014-11-10 builds Created: 10/Nov/14  Updated: 23/Mar/15  Resolved: 13/Nov/14

Status: Closed
Project: Core Server
Component/s: Performance
Affects Version/s: 2.8.0-rc0
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: David Daly Assignee: Unassigned
Resolution: Done Votes: 0
Labels: 28qa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

Mongo-sim table_scan workload. States 12 and 15. In Comments.

Participants:

 Description   

Mongo-sim based microbenchmarks that update a hot document (i.e., all updating same document) show significant performance degradation today. 10 threads trying to update the same document.

Throughput for the two tests
12: 2489 2458 2386 419 375
15: 45328 43026 43455 13261 5895
12 is a table scan based, and 15 is index scan based. Last two numbers are for builds from today(4ab029e3 and c686f5a19 respectively). First three from last week.



 Comments   
Comment by David Daly [ 13/Nov/14 ]

Closing this one since the performance regression was relative to a incorrect behavior.

Comment by David Daly [ 13/Nov/14 ]

asya, it looks like you are right. I tried out before and after the changset for SERVER-15998 on my laptop and that accounts for most of the performance hit. I'm going to rerun on the original linux box to confirm.

Comment by Asya Kamsky [ 11/Nov/14 ]

Couldn't this be due to incorrect behavior on conflict on write fixed in SERVER-15998?

Comment by David Daly [ 11/Nov/14 ]

Updated results show a big improvement with version 92d967 for some of the mongo-perf simple_mms_update particularly single threaded, but still trailing at 10 threads.

Update.MmsIncDeepDistinctPath2 and Update.MmsIncDeepDistinctPath3 show the biggest remaining mulithreaded drop.

MongoDB shell version: 2.7.9-pre-
connecting to: 127.0.0.1:27017/test
db version: 2.7.9-pre-
92d967e8cd4d32f3b3348c89320f238df47a1454

load('util/utils.js')

load('testcases/simple_update_mms.js')

mongoPerfRunTests([1, 2, 10], 1, 1, 5, 1, '', 'sanity', 'localhost', '27017', '2014-11-11 09:13:04.439825', 0,

{"writeCmdMode": "true", "writeConcernW": 0, "safeGLE": "false", "writeConcernJ": "false"}

, {"server_git_hash": "92d967e8cd4d32f3b3348c89320f238df47a1454", "server_storage_engine": "mmapv1", "harness": {"git_hash": "unknown", "client":

{"git_hash": "92d967e8cd4d32f3b3348c89320f238df47a1454", "version": "2.7.9-pre-", "name": "mongo shell"}

, "name": "mongo-perf", "version": "unknown"}, "server_version": "2.7.9-pre-"});

Update.MmsIncShallow1
1 13519.945729823476
2 26124.8870798104
10 30881.237761253247
Update.MmsIncShallow2
1 13085.210060740188
2 25277.958548107814
10 30171.964700351145
Update.MmsIncDeep1
1 12550.573280508463
2 23841.99414059805
10 27023.686821843086
Update.MmsIncDeepSharedPath2
1 11983.706271301622
2 23534.86881768717
10 25074.967448476244
Update.MmsIncDeepSharedPath3
1 11560.046262584365
2 22816.050852284545
10 23646.00504220401
Update.MmsIncDeepDistinctPath2
1 11768.289256246278
2 21762.06858346463
10 22827.32463755248
Update.MmsIncDeepDistinctPath3
1 11105.09582864938
2 21316.883374014917
10 21449.972893572627
Finished Testing.

Comment by Chung-yen Chang [ 11/Nov/14 ]

I reviewed the mongo-perf data from the last couple weeks and the interesting about the MMS related tests that David reported above is that the performance of those operations saw a big bump from Nov. 02 to Nov. 03. The between Nov. 06 and Nov. 07, we had this drop and the performance for those are basically back to the same level as before the big jump on the 3rd. Could this be something checked in around the 3rd got reverted?

Comment by David Daly [ 11/Nov/14 ]

Can be reproduced with mongo-perf simple_update_mms.js tests.

Results from Friday build:
MongoDB shell version: 2.7.9-pre-
connecting to: 127.0.0.1:27017/test
db version: 2.7.9-pre-
e1fa60e4d8b3f525575b4602b208f2a1ef353735

load('util/utils.js')

load('testcases/simple_update_mms.js')

mongoPerfRunTests([1, 2, 10], 1, 1, 5, 1, '', 'sanity', 'localhost', '27017', '2014-11-10 20:26:34.117036', 0,

{"writeCmdMode": "true", "writeConcernW": 0, "safeGLE": "false", "writeConcernJ": "false"}

, {"server_git_hash": "e1fa60e4d8b3f525575b4602b208f2a1ef353735", "server_storage_engine": "wiredtiger", "harness": {"git_hash": "unknown", "client":

{"git_hash": "e1fa60e4d8b3f525575b4602b208f2a1ef353735", "version": "2.7.9-pre-", "name": "mongo shell"}

, "name": "mongo-perf", "version": "unknown"}, "server_version": "2.7.9-pre-"});

Update.MmsIncShallow1
1 5387.869405686705
2 10346.68059663405
10 36773.01260564449
Update.MmsIncShallow2
1 2704.0739901520587
2 11070.846304455816
10 35303.44937804129
Update.MmsIncDeep1
1 4145.163948786982
2 12022.550782257757
10 31695.022255211286
Update.MmsIncDeepSharedPath2
1 5400.731888185331
2 8730.902338566702
10 33936.71194292096
Update.MmsIncDeepSharedPath3
1 3465.989267852515
2 10754.621543982212
10 19927.94831974583
Update.MmsIncDeepDistinctPath2
1 4602.126997739766
2 7208.705300085804
10 35623.83609683674
Update.MmsIncDeepDistinctPath3
1 2699.4688058160373
2 11984.627830914062
10 34101.45363440599
Finished Testing.

Today's builds:
MongoDB shell version: 2.7.9-pre-
connecting to: 127.0.0.1:27017/test
db version: 2.7.9-pre-
4ab029e3e68ea5f7aa89b90a47fd814d17b142bf

load('util/utils.js')

load('testcases/simple_update_mms.js')

mongoPerfRunTests([1, 2, 10], 1, 1, 5, 1, '', 'sanity', 'localhost', '27017', '2014-11-10 20:30:12.102994', 0,

{"writeCmdMode": "true", "writeConcernW": 0, "safeGLE": "false", "writeConcernJ": "false"}

, {"server_git_hash": "4ab029e3e68ea5f7aa89b90a47fd814d17b142bf", "server_storage_engine": "wiredtiger", "harness": {"git_hash": "unknown", "client":

{"git_hash": "4ab029e3e68ea5f7aa89b90a47fd814d17b142bf", "version": "2.7.9-pre-", "name": "mongo shell"}

, "name": "mongo-perf", "version": "unknown"}, "server_version": "2.7.9-pre-"});

Update.MmsIncShallow1
1 6737.886014511724
2 6622.532449209087
10 3845.7036893445347
Update.MmsIncShallow2
1 6185.674633514277
2 5531.5420607442375
10 3545.3450282721888
Update.MmsIncDeep1
1 4494.897529236861
2 5788.798466177198
10 6023.415849800333
Update.MmsIncDeepSharedPath2
1 1714.1266353800058
2 5876.233323040642
10 4449.179821800881
Update.MmsIncDeepSharedPath3
1 4524.605441965996
2 4377.35683684745
10 6266.528814789831
Update.MmsIncDeepDistinctPath2
1 5030.415749300684
2 7240.523792040764
10 3401.797353402449
Update.MmsIncDeepDistinctPath3
1 2694.7806223755783
2 6330.950570454653
10 4082.4566386183733
Finished Testing.

MongoDB shell version: 2.7.9-pre-
connecting to: 127.0.0.1:27017/test
db version: 2.7.9-pre-
c686f5a1949339ba9390e6efc01ef8421867c692

load('util/utils.js')

load('testcases/simple_update_mms.js')

mongoPerfRunTests([1, 2, 10], 1, 1, 5, 1, '', 'sanity', 'localhost', '27017', '2014-11-10 20:35:48.689789', 0,

{"writeCmdMode": "true", "writeConcernW": 0, "safeGLE": "false", "writeConcernJ": "false"}

, {"server_git_hash": "c686f5a1949339ba9390e6efc01ef8421867c692", "server_storage_engine": "wiredtiger", "harness": {"git_hash": "unknown", "client":

{"git_hash": "c686f5a1949339ba9390e6efc01ef8421867c692", "version": "2.7.9-pre-", "name": "mongo shell"}

, "name": "mongo-perf", "version": "unknown"}, "server_version": "2.7.9-pre-"});

Update.MmsIncShallow1
1 7302.995414054061
2 5745.3124809960045
10 4858.553712321445
Update.MmsIncShallow2
1 5636.545757234148
2 1672.2907342530657
10 6150.579877796645
Update.MmsIncDeep1
1 1830.9661278211604
2 6794.260091789858
10 5677.685865923321
Update.MmsIncDeepSharedPath2
1 4837.431125896663
2 6657.849480180749
10 3748.3712412403197
Update.MmsIncDeepSharedPath3
1 6028.324246459469
2 2694.6507163503143
10 5943.981827708826
Update.MmsIncDeepDistinctPath2
1 5648.794416560891
2 3732.0550127295746
10 5721.351233289057
Update.MmsIncDeepDistinctPath3
1 6010.587507593661
2 3522.005765209036
10 5613.250286825488
Finished Testing.

Comment by David Daly [ 10/Nov/14 ]

May be related to SERVER-16056 ?

Generated at Thu Feb 08 03:39:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.